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DEVELOPING  INTERCULTURAL  ADAPTABILITY  IN  THE  WARFIGHTER: 
A  WORKSHOP  ON  CULTURAL  TRAINING  AND  EDUCATION 

EXECUTIVE  SUMMARY 


Research  Requirement: 

Understanding  and  navigating  the  sociocultural  terrain  have  not  historically  been  recognized  as 
core  warfighter  competencies,  but  have  proved  to  be  an  unavoidable  component  of  military 
operations  in  recent  years.  From  humanitarian  relief  operations  to  counter-insurgency  operations 
in  Iraq  and  Afghanistan,  culture  has  emerged  as  a  central  consideration  for  both  general-purpose 
forces  and  specialists.  Cultural  training  and  education  can  provide  the  capabilities  needed  to 
confront  the  cultural  complexity  characteristic  of  the  current  and  future  missions.  Due  to  the 
wide  array  of  efforts  in  this  area,  opportunities  are  needed  to  coordinate  efforts,  share  successes 
and  lessons  learned,  and  identify  common  challenges. 

Procedure: 

A  two-day  workshop  focused  on  the  instructional  design  process  as  applied  to  the  development 
of  cultural  training  and  education.  Participants  were  approximately  130  representatives  from 
government,  industry,  and  academia  who  are  actively  involved  in  planning,  developing,  or 
delivering  cultural  training  and  education  or  in  conducting  research  in  those  areas.  Participants 
attended  plenary  session  presentations  by  the  Services’  culture  centers  and  other  government 
programs  addressing  cultural  capabilities.  The  plenary  session  presentations  described  current 
and  emerging  approaches,  as  well  as  highlighted  gaps  and  issues  needing  further  attention.  Four 
breakout  sessions  examined  aspects  of  the  instructional  design  process  in  greater  detail,  with 
parallel  tracks  on  analysis,  design,  development,  and  implementation  and  evaluation. 

Findings: 

Research  gaps  needing  further  investment  were  identified  in  six  primary  areas: 

1)  Cultural  performance  requirements  analysis.  Cognitive  task  analysis  and  other  research 
should  be  conducted  to  determine  empirically  what  Service  members  need  to  do  or  know 
regarding  culture  to  perform  their  missions. 

2)  Learner  motivation,  cross-cultural  competence,  and  development.  Models  of  cultural 
expertise  and  cultural  skill  acquisition  are  greatly  needed  to  inform  instructional  design. 
In  addition,  research  is  needed  to  determine  which  instructional  methods  best  support 
learning  and  motivation.  The  roles  of  fun,  fidelity,  and  feedback  in  instructional  design 
are  areas  of  particular  interest. 

3)  Development  and  validation  of  instructional  sociocultural  content.  Identification  of  and 
taxonomies  for  the  cultural  information  Service  members  need  for  their  varied  missions 
and  role  is  needed  to  guide  training  content.  Methods  to  more  efficiently  gather,  access, 
and  validate  such  information  are  also  needed. 
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4)  Flexible  instructional  solutions.  Greater  flexibility  and  authorability  of  instructional 
technology  are  in  high  demand.  Further  advances  are  needed  to  develop  instructional 
solutions  that  provide  tailoring  to  individual  learners’  needs  and  offer  instructors  and 
trainers  flexibility  to  change  scenarios,  conditions,  or  cultural  context. 

5)  Methods  and  metrics  for  training  evaluation.  Evaluation  of  cultural  training  and 
simulation  is  greatly  needed,  yet  metrics  and  methods  to  evaluate  beyond  simple  user 
reaction  are  not  in  use.  Research  is  needed  to  provide  methods  to  evaluate  the  impact  of 
cultural  training  on  the  learning  outcomes  identified  by  the  Services  as  important. 

6)  Continuing  opportunities  for  exchange  and  collaboration.  Due  to  the  interdisciplinary  and 
inter-Service  nature  of  the  challenges  for  cultural  training  and  education,  researchers, 
training  developers,  and  practitioners  would  benefit  from  continuing  opportunities  to 
collaborate  and  disseminate  findings  and  solutions. 

Utilization  and  Dissemination  of  Findings: 

These  findings  can  be  used  to  inform  future  directions  for  the  training  portion  of  the  Human, 
Social,  Culture,  and  Behavior  Modeling  Program,  as  well  as  other  research  and  development 
programs  addressing  cultural  training  and  education. 
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Developing  Intercultural  Adaptability  in  the  Warfighter: 

A  Workshop  on  Cultural  Training  and  Education 

INTRODUCTION 

Understanding  and  navigating  the  sociocultural  terrain  have  not  historically  been 
recognized  as  core  warfighter  competencies,  but  have  proved  to  be  an  unavoidable  component  of 
military  operations  in  recent  years.  From  humanitarian  relief  operations  to  counter-insurgency 
operations  in  Iraq  and  Afghanistan,  culture  has  emerged  as  a  central  consideration  for  both 
general-purpose  forces  and  specialists.  Cultural  training  and  education  can  provide  the 
capabilities  needed  to  confront  the  cultural  complexity  characteristic  of  the  current  and  future 
missions.  Across  the  Department  of  Defense,  policies  and  programs  have  proliferated  to  develop 
these  capabilities.  The  challenge  now  is  to  identify  and  further  develop  solutions  that  effectively 
build  them. 

Due  to  the  wide  array  of  efforts  in  this  area,  opportunities  are  needed  to  coordinate 
efforts,  share  successes  and  lessons  learned,  and  identify  common  challenges.  This  workshop 
provided  such  an  opportunity.  The  focus  of  this  two-day  event  was  the  instructional  design 
process  as  applied  to  the  development  of  cultural  training  and  education.  Participants  were 
approximately  130  representatives  from  government,  industry,  and  academia  who  are  actively 
involved  in  planning,  developing,  or  delivering  cultural  training  and  education  or  in  conducting 
research  in  those  areas.  Participants  attended  plenary  session  presentations  by  the  Services’ 
culture  centers  and  other  programs  addressing  cultural  capabilities  (see  Appendix  A  for 
workshop  program).  The  plenary  session  presentations  described  current  and  emerging 
approaches,  as  well  as  highlighted  gaps  and  issues  needing  further  attention.  Four  breakout 
sessions  examined  aspects  of  the  instructional  design  process  in  greater  detail,  with  parallel 
tracks  on  analysis,  design,  development,  and  implementation  and  evaluation. 

This  report  will  present  summary  of  the  presentations  and  discussions  from  the 
workshop.  First,  a  summary  of  the  Services’  current  approaches  to  cultural  training  and 
education  will  be  presented.  Next,  other  themes  from  the  plenary  sessions  will  be  identified, 
followed  by  summaries  and  recommendations  from  each  of  the  breakout  session  tracks.  Finally, 
an  overall  set  of  gaps  and  recommendations  will  be  outlined  for  future  research. 

As  Martin  Bushika  noted  in  his  presentation,  we  are  at  the  “end  of  the  beginning”  of 
incorporating  culture  into  military  training  and  education.  All  of  the  Services  have  implemented 
programs  to  better  prepare  their  personnel  for  the  cultural  environments  of  current  conflicts.  This 
point  is  a  good  time  to  assess  collective  progress  in  this  domain  and  to  determine  where  further 
research  and  development  are  needed. 
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Current  Approaches,  Programs,  and  Challenges 


Programs  and  Perspectives  from  the  Services 

All  of  the  Services  have  a  culture-general  component  and  a  culture-specific  component  to 
cultural  training  and  education  for  general-purpose  forces,  though  they  differ  on  the  relative 
emphasis  on  each.  For  example,  the  Air  Force  strategy  identifies  cross-cultural  competence  as 
the  primary  enabler  for  Ainnen  to  communicate,  relate,  and  negotiate  across  cultures, 
supplemented  by  language  and  regional  culture  (2009).  David  Ott  of  the  U.S.  Army  Training  and 
Doctrine  Command  noted  that  the  Army  approach  is  “big  C,  little  1,”  placing  relatively  more 
emphasis  on  culture  than  on  language  for  the  general-purpose  forces  (Department  of  the  Anny, 
2009).  The  Marine  Corps  includes  a  culture-general  component  with  its  “operational  culture” 
concept  and  teaching  of  interpersonal  skills  in  advisor  training,  but  emphasizes  regional  studies 
both  in  its  pre-deployment  training  and  in  the  career  development  cycle  with  its  Career  Marine 
Regional  Studies  Program.  The  Navy  also  emphasizes  the  culture-specific  component, 
highlighting  language  and  region  initiatives  in  its  strategy  (2008). 

Speakers  from  the  Services’  culture  centers  and  programs  identified  challenges  in 
developing  and  implementing  appropriate  solutions.  David  Ott  noted  constraints  on  time 
available  for  cultural  training.  Training  schedules  that  are  already  tightly  packed  with  essential 
tasks  demand  cultural  training  solutions  that  are  effective  and  efficient.  Paula  Hohnes-Eber 
discussed  the  importance  of  understanding  and  tailoring  to  the  audience.  From  her  experiences 
teaching  operational  culture  to  Marines,  she  observed  the  need  to  make  cultural  training  relevant 
and  immediately  applicable  to  Marine  Corps  tasks.  She  also  noted  the  need  for  collective 
training. 

Both  Rob  Sands  (Air  Force  Culture  and  Language  Center)  and  David  Ott  noted  the  role 
of  Service  differences  in  understanding  the  training  requirements.  For  example,  whereas  ground 
forces  tend  to  deploy  as  units,  Airmen  deploy  as  individuals,  which  generates  some  differences 
in  how  cultural  training  should  be  delivered.  David  Brand  (John  F.  Kennedy  Special  Warfare 
Center  and  School)  identified  the  challenge  of  determining  what  success  is  in  cultural  training 
and  education.  In  other  words,  what  does  a  culturally  competent  Soldier  look  like?  What  are  the 
skills  and  knowledge  needed  at  intermediate  or  advanced  levels?  Once  those  characteristics  are 
identified,  assessment  tools  are  needed  to  measure  them,  a  challenge  that  the  Air  Force  Culture 
and  Language  Center  is  also  currently  facing. 

Speakers’  comments  included  many  suggestions  for  developing  or  improving  the  cultural 
component  of  training.  LTC  Marc  Kortenray  (U.S.  Army  Intelligence  Center,  Training  Division) 
emphasized  the  use  of  partnerships  with  other  sectors  to  help  address  these  challenges. 
Establishing  relationships  with  academic  institutions  would  help  provide  access  to  regional 
experts  who  can  contribute  training  content.  Collaboration  and  communication  with  the 
Combatant  Commands  would  also  help  ensure  that  training  meets  operational  requirements. 

Martin  Bushika  noted  the  utility  of  building  platfonns  or  architectures  that  can  be  used 
for  different  applications  in  order  to  enhance  affordability.  He  also  advocated  the  development  of 
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portable  solutions.  For  example,  Web-based  tools  can  facilitate  tracking  an  individual’s  progress 
so  that  the  information  is  available  to  unit  leadership.  In  addition,  training  and  practice  tools  for 
handheld  devices  can  provide  timely  opportunities  to  reinforce  learning  from  other  training 
exercises,  such  as  Mojave  Viper,  and  can  help  sustain  capabilities  like  language  skills  that  are 
subject  to  decay. 

Perspectives  on  Instructional  Technologies  for  Cultural  Learning 

Plenary  sessions  included  presentations  from  two  experts  in  technology-based  instruction, 
Dr.  Richard  Clark  (University  of  Southern  California)  and  Dr.  Dexter  Fletcher  (Institute  of 
Defense  Analyses).  These  experts  helped  frame  the  issues  of  applying  instructional  technologies 
to  cultural  training. 

Dr.  Clark  discussed  five  principles  to  consider  when  developing  cultural  instruction.First, 
culture  is  largely  automated  and  unconscious.  Identifying  the  important  aspects  of  culture  to 
address  in  training  is  therefore  difficult,  but  can  be  improved  by  the  use  of  cognitive  task 
analysis.  Cognitive  task  analysis  can  help  access  the  deep  cultural  knowledge  of  experts.  Second, 
Dr.  Clark  recommended  that  cultural  instruction  in  the  military  first  teach  about  our  own 
culture(s).  Understanding  one’s  own  cultural  context  and  influences  provides  the  foundation  for 
understanding  other  cultures.  The  goal  is  to  achieve  cultural  understanding,  not  to  insist  on 
agreement  with  other  cultural  beliefs  and  values.  The  training  audience  will  be  more  receptive  if 
they  realize  that  the  goal  is  to  enable  them  to  better  do  their  jobs.  Third,  culture  dictates 
motivation,  and  motivation  is  a  primary  consideration  in  developing  training  and  education. 

What  gets  people  to  start  an  activity,  why  do  they  persist,  and  how  do  we  get  them  to  invest  time 
and  effort  in  it?  Fourth,  learn  “how”  and  “why”  before  adapting.  People  need  to  know  how  to  do 
something  and  why  it  will  help  them.  We  can’t  just  convey  the  “what”  and  expect  it  to  transfer. 
Fifth,  varying  practice  is  needed.  Dealing  with  foreign  cultures  requires  adaptive  performance. 
Learning  a  routine  response  first,  then  encountering  varied  conditions  and  novel  circumstances 
helps  to  build  adaptability.  Avoiding  cultural  stereotypes  will  help  enable  adaptability  so  that 
Service  members  are  prepared  for  the  variation  among  individuals  and  subcultures  that  they  will 
inevitably  encounter. 

Dr.  Fletcher  emphasized  the  need  for  training  evaluation  and  cautioned  workshop 
participants  to  pay  attention  to  cost.  Return  on  investment  is  often  difficult  to  quantify  in  the 
defense  context,  but  can  be  addressed  by  looking  at  the  cost  of  training  in  terms  of  personnel 
time  and  the  investment  in  developing  (and  maintaining)  the  training  resources.  Training  is  a 
means  to  achieve  greater  operational  effectiveness,  not  an  end  in  itself.  Dr.  Fletcher  called 
attention  to  the  issue  of  requirements  and  identifying  the  tactically  sufficient  levels  of  skill 
needed  to  accomplish  the  mission.  He  also  cited  the  advantages  of  instructional  technologies  in 
reducing  the  time  needed  to  train  and  the  potential  for  providing  individualized  instruction. 

These  advantages  can  be  accomplished  by  taking  advantage  of  the  learner’s  prior  knowledge.  Dr. 
Fletcher  encouraged  the  audience  to  think  of  instructional  design  as  an  engineering  discipline, 
rather  than  as  an  art  or  a  science. 


3 


Themes  and  Recommendations  from  Breakout  Sessions 


This  section  presents  summaries  and  recommendations  from  the  four  tracks  of  breakout 
sessions:  analysis,  design,  development,  and  implementation  and  evaluation.  In  some  sessions, 
discussions  extended  beyond  training,  education,  and  research  and  tended  to  address  relevant, 
but  broader  concerns,  such  as  personnel  selection  and  management.  We  have  attempted  to 
include  that  material  in  the  session  themes  and  summaries,  but  restricted  recommendations  to 
issues  for  cultural  training,  education,  and  research. 

Analysis  Track  Summary 

Mr.  Marc  Hill,  of  the  Defense  Language  Office,  and  Mr.  Doug  Nelson,  of  Kinection, 
opened  the  first  session,  Toward  a  Shared  Competency  Map.  They  began  with  a  brief 
background  on  the  Defense  Language  Office,  which  is  tasked  to  create  force  development  policy 
on  regional  expertise,  culture,  and  foreign  language  that  will  be  used  throughout  the  Department 
of  Defense.  Mr.  Nelson  presented  on  the  need  for  a  shared  map  of  competencies  that  all  Service 
members  need  to  work  effectively  in  foreign  cultures.  A  shared  competency  map  would  enable 
different  instructional  developers  and  vendors  to  use  a  common  framework. 

The  analysis  track  then  divided  into  five  smaller  groups  which  were  asked  to  look  at  the 
potential  benefits  of  a  shared  map  and  possible  obstacles  to  developing  such  a  map.  Groups 
were  asked  to  define  role-based  perfonnance  objectives  that  support  the  competencies  and  to 
define  knowledge,  skills,  and  abilities  (KSAs)  that  support  the  perfonnance  objectives.  Groups 
split  up  depending  on  their  organization:  vendors  (2),  agencies,  training  organizations,  and  nay¬ 
sayers,  who  expressed  skepticism  about  the  shared  map  concept,  and  then  discussed  these  issues 
for  30  minutes.  Each  group  then  out-briefed  what  they  had  discussed  to  the  analysis  track. 

The  first  vendor  group  discussed  the  need  for  a  core  group  comprised  of  government, 
military,  industry,  and  academia.  Each  of  these  groups  would  gather  information  from  their 
specific  community,  and  the  group  would  synthesize  the  competencies.  The  second  vendor 
group  focused  on  processes  for  performing  missions;  first  look  at  what  is  going  on,  then  use  the 
knowledge  one  has,  come  up  with  objectives,  and  then  execute  the  plan.  They  also  discussed 
planning  for  social  outcomes  and  military  objectives  using  social  reasoning.  The  nay-sayer 
group  focused  on  competencies,  noting  the  need  to  understand  linguistic  pragmatics  and  to 
understand  verbal/non-verbal  queues.  They  noted  that  it  is  important  to  know  the  costs  and 
investments  of  training  systems.  This  group  also  questioned  what  the  private  sector  and  business 
schools  are  developing  and  discussed  the  need  for  culture  to  have  the  same  level  of  importance 
as  other  military  priorities.  The  training  group  focused  on  the  need  for  a  foundational  doctrine 
base  to  show  the  importance  of  culture  and  for  sequencing  as  a  foundation  for  training  and 
education.  Self-awareness  needs  to  be  integrated  as  well  and  should  be  a  leadership  exercise. 
Finally,  the  agency  group  had  a  spirited  dialogue  and  detennined  that  it  is  important  to  map 
specific  competencies.  They  discussed  the  need  to  define  who  would  benefit  most  and  who 
would  find  it  most  problematic.  They  also  discussed  participation  in  this  area  by  the  US 
Department  of  State. 
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The  second  session,  Motivational  and  Personality  Factors  Affecting  Cultural  Learning, 
was  opened  by  Dr.  Joan  Johnston,  from  the  Naval  Air  Warfare  Center,  Training  Systems 
Division  who  gave  a  short  presentation  on  cross-cultural  competence  and  mission  effectiveness. 
She  discussed  personality  characteristics  that  make  people  most  amenable  to  cross-cultural 
competence.  Dr.  Johnston  discussed  the  three  components  of  cross-cultural  competence: 
cognitive,  behavioral,  and  attitudinal.  She  also  discussed  research  findings  citing  the  importance 
of  emotional  regulation. 

The  rest  of  the  session  included  a  presentation  entitled  Necessary  Antecedents  for 
Intercultural  Learning  by  Dr.  Paula  Caligiuri,  Professor  of  Human  Resource  Management  at 
Rutgers  University  and  included  an  open  dialogue  on  cultural  agility,  which  looked  at  current 
gaps  and  future  training  possibilities.  The  questions  and  answers  portion  focused  on  the  hard¬ 
wiring  of  a  person  versus  the  ability  to  train  the  individual,  the  significance  of  motivational 
factors  to  leam  about  foreign  cultures,  an  individual’s  background  and  how  it  influences  them  in 
the  future,  the  value  of  inherent  personality,  how  to  best  understand  and  evaluate  one’s  own 
culture,  and  potential  methods  to  evaluate  and  select  people  who  will  best  adapt  to  foreign 
culture  and  language. 

Dr.  Caligiuri  described  cultural  agility,  which  is  the  ability  to  work  with  people  from 
different  cultures  and  not  solely  about  adaptation.  Following  a  question  asking  how  the  hard¬ 
wiring  of  an  individual  affects  their  ability  to  be  agile  in  a  foreign  culture,  Dr.  Caligiuri  noted  it 
is  difficult  to  adapt  if  a  person  does  not  possess  certain  characteristics  to  function  well  in  high- 
level  peer-to-peer  interactions.  This  led  to  a  group  discussion  on  the  selection  process  for 
Special  Operations  Forces  (SOF)  that  are  based  on  certain  characteristic  traits  that  make  them 
more  likely  to  succeed,  and  how  powerful  self-selection  is.  Dr.  Caligiuri  also  discussed  how 
people  who  perfonn  well  in  one  culture  tend  to  do  well  in  other  cultures  as  well,  emphasizing 
culture-general  abillities.  She  also  discussed  the  importance  of  motivation,  noting  in  the 
corporate  realm,  she  looks  for  people  who  have  openness  to  foreign  cultures,  are  emotionally 
stable  and  who  are  sociable. 

One  theme  of  the  discussion  focused  on  a  necessary  shift  within  the  military  that 
demonstrates  the  importance  of  culture,  including  the  need  to  promote  its  humanitarian  side  in 
addition  to  its  more  conventional  side.  Culture  was  framed  as  a  life-saving  skill.  One  difficulty 
the  group  focused  on  is  that  a  prior  negative  experience  in  a  foreign  culture  (such  as  being  in  Iraq 
during  high-intensity  kinetic  warfare)  can  have  implications  on  one’s  motivation  to  respect  a 
culture  on  future  deployments.  It  is  also  important  that  behavior,  both  negative  and  positive, 
correlate  with  an  outcome,  such  as  being  rewarded  for  good  behavior.  Additionally,  attendees 
noted  that  the  climate  of  the  organization  and  ranking  officers  has  a  great  impact  on  the  culture 
of  a  unit  and  the  value  placed  on  intercultural  aspects  of  the  mission.  The  group  discussed  the 
dilemma  of  including  more  training  hours  for  culture  into  existing  military  education  which  is 
already  compact. 

The  group  also  focused  on  an  individual’s  personality,  including  the  difference  between 
extroverts,  who  tend  to  be  more  culturally  agile,  and  introverts,  who  may  have  more  difficulty  in 
knowing  what  to  monitor  in  a  foreign  culture.  The  group  also  discussed  early-life  experiences 
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and  whether  this  leads  to  doing  well  in  intercultural  settings  later  in  life.  In  one  study,  bilingual 
children  were  more  effective  working  in  bilingual  cultures  later  in  life.  Dr.  Caligiuri  said  that  the 
more  languages  a  person  speaks,  the  more  agile  he  is  in  a  foreign  territory  regardless  of  what 
culture  it  is.  Next,  they  discussed  the  importance  of  learning  about  one’s  own  culture  before 
learning  about  another.  The  military  also  has  the  ability  to  shape  behavior  and  should  do  so.  An 
attitude  shift  might  have  a  big  impact,  and,  over  time,  values  and  attitudes  might  shape  behavior 
in  a  positive  way.  The  group  discussed  how  nowadays,  people  self-select  to  join  the  military 
because  they  identify  with  certain  aspects  of  the  military  which  pushed  them  to  join.  Perhaps  it 
is  necessary  to  identify  what  aspects  make  people  join  and  to  then  build  upon  those  areas. 

Dr.  Johnston  then  noted  that  when  we  have  representation  from  different  places  that  want 
to  establish  different  dimensions  of  competence,  and  many  are  personality  and  motivationally 
related  and  there  are  models  to  help  predict  success  and  learning,  these  things  should  help  us 
figure  out  how  we  are  going  to  use  our  resources  to  build  training,  education,  and  monitoring 
systems. 

The  group  discussed  many  things,  including  the  need  for  people  to  alter  their  perception 
of  the  traditional  military  mindset  of,  for  example,  blowing  things  up,  to  a  more  humanitarian 
side.  This  shift  in  messaging  will  take  a  long  time,  and  should  begin  with  cultural  awareness 
being  taught  in  boot  camps.  Thus,  Soldiers  need  to  be  competent  in  both  the  traditional  military 
as  well  as  more  cultural  humanitarian  skills.  Therefore,  cultural  education  should  be  done  in 
conjunction  with  more  traditional  teaching.  If  Service  members  understand  how  cultural  training 
will  help  them  perfonn  better,  the  training  will  be  more  effective.  Talking  to  Service  members 
about  attitudes  and  cultural  values  along  the  rank  structure  can  help.  Military  personnel  who 
were  involved  in  kinetic  operations  have  a  difficult  time  being  friendly  to  foreigners  following  an 
attack.  The  group  discussed  the  need  to  mitigate  these  experiences.  The  conclusion  was  to  show 
utility  for  whatever  is  being  taught  to  Soldiers,  so  that  they  will  be  able  to  tie  the  utility  and 
success  to  what  they  are  being  taught.  This  led  to  a  conversation  on  the  linkage  between 
behavior  and  desired  outcome.  Measuring  and  rewarding  a  positive  outcome  is  very  important. 

In  addition,  the  organizational  climate  is  also  of  importance.  If  a  leader  finds  cultural  agility 
important,  it  could  change  the  entire  unit’s  opinion  of  culture. 

Session  three,  Socio-cultural  Aspects  of  Mission  Performance  in  Specific  Roles  and  MOS, 
began  with  a  presentation  by  Dr.  Michelle  Ramsden  Zbylut  (U.S.  Army  Research  Institute,  Fort 
Leavenworth).  She  discussed  the  need  to  focus  on  leadership  development  and  methods  for 
looking  at  mission  performance  in  regards  to  culture.  Specifically,  the  military  should  identify 
methods  for  looking  at  cultural  capabilities  needed  for  different  missions  and  effective 
performance  and  the  role  of  training  versus  other  solutions  toward  achieving  cultural  effectives. 

Dr.  Zbylut’ s  group  applies  task  analysis  methodology  to  examine  the  soft  human  skills 
and  activities  that  military  advisors  focus  on.  It  is  important  to  know  how  to  focus  cultural 
training  for  advisors,  given  the  lack  of  time  allocated  to  cultural  training.  She  described  a  survey 
conducted  by  her  team,  in  which  returning  advisors  prioritized  behaviors  and  skills  relative  to 
one  another  to  find  what  should  be  addressed  in  training.  This  survey  identified  advisors’ 
perceptions  of  how  frequent  and  important  each  behavior  was.  Although  it  provided  a  good  way 
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to  understand  the  performance  domain,  there  was  no  perspective  given  from  neither  locals  nor 
experts.  It  was  noted  that  success  was  very  difficult  to  look  at  in  this  context,  but  that  behaviors 
were  prioritized  relative  to  other  behaviors. 

The  second  part  of  this  session  was  led  by  Dr.  Karol  Ross  (Cognitive  Performance 
Group)  on  Methods  for  Understanding  Intercultural  Aspects  of  Military  Job  Performance. 
Methods  depend  on  goals  and  it  is  important  to  always  state  goals  very  clearly.  She  focused  on 
cross-cultural  competence  as  an  emerging  area  of  military  expertise.  Dr.  Ross  presented  different 
survey  examples  done  in  this  field.  In  one  example,  her  team  used  a  team-ranking  task  that  Mr. 
Mike  McCloskey  developed.  Rankings  ranged  from  not  competent  to  culturally  competent,  and 
interviewees  were  asked  to  categorize  team  members  along  this  continuum.  The  researchers  also 
used  critical  incident  interviews,  task  diagrams,  team-ranking  tasks,  and  document  review  to  the 
role  of  culture  in  Air  Force  roles  and  missions.  Results  ranged  from  NCO  through  officer 
personnel,  all  with  recent  foreign  contact.  Dr.  Ross  found  that  the  Air  Force  put  a  greater 
emphasis  on  language  than  the  Army  did.  She  then  showed  a  task  diagram  and  described  the 
way  it  works.  Its  goal  is  to  break  a  person’s  tendencies  to  say  what  they  should  do,  and  to 
capture  what  they  actually  do.  Challenges  to  the  analysis  of  intercultural  aspects  of  specific  roles 
include  difficulty  in  distinguishing  the  culture-general  aspects  of  performance  from  the  region- 
specific  and  in  distinguishing  cultural  competence  from  tactical  competence. 

The  group  discussed  methods  previously  used  as  well  as  knowing  when  to  use  the 
appropriate  method.  For  example,  they  discussed  the  need  to  find  a  baseline  and  ways  to 
determine  perfonnance  in  tenns  of  effectiveness,  both  for  experts  and  non-experts.  Building  on 
earlier  discussions,  the  group  focused  on  the  concept  of  cross-cultural  competence  and  debated 
whether  it  is  a  value  needing  to  be  instilled  in  every  Service  member,  or  rather  a  set  of  KSAs  that 
need  to  be  acquired  through  training.  The  group  concluded  it  should  be  developed  (and 
evaluated)  through  training.  Additionally,  they  noted  that  cross-cultural  competence  is  multi¬ 
faceted  and  its  component  skills  must  be  taught  in  military  training.  They  also  continued  to 
discuss  the  importance  of  Soldiers  understanding  why  and  where  they  are  going  if  it  is  for  an 
extended  period  of  time.  It  is  a  long-term  professional  development  process  that  includes 
military  culture,  training,  and  continuing  education.  If  a  person  is  going  abroad  to  perform  a 
technical  task,  then  training  is  sufficient,  however;  if  the  person  is  going  for  a  longer  period  of 
time,  selecting  personnel  with  higher  levels  of  cross-cultural  competence  may  be  more 
appropriate  than  training.  Continuing  the  discussion  of  selection,  the  group  recommended 
recruiting  globally-minded  individuals  using  advertising  methods  that  show  the  military’s 
cultural  side.  Furthermore,  the  military  must  begin  now  to  focus  on  future  missions  and  new 
countries  and  regions  so  that  it  can  best  determine  what  the  performance  domain  might  look  like 
in  potential  future  missions. 

There  is  currently  a  gap  in  demonstrating  a  direct  linkage  between  cross-cultural  skills 
and  the  success  of  a  mission,  and  whether  knowing  about  culture  improves  the  effectiveness  of  a 
mission.  Part  of  this  gap  includes  performance  measurement.  The  group  discussed  discourse 
analysis  as  one  method  to  address  this  gap.  Take  examples  of  language  in  use  that  can  be  in  any 
form  -  documentary,  recording,  etc.  -  and  analyze  the  structural  components  of  the  language  and 
what  is  implied  in  the  background  with  language  use.  This  method  could  also  be  used  in  training 


7 


evaluation,  with  both  a  pre-  and  post-test  to  see  how  actual  communicative  interpretations  shift 
over  time.  Word  choices  are  important  and  may  reflect  an  individual’s  level  of  understanding  or 
competence. 

This  session  closed  by  discussing  that  the  Services  do  not  have  a  good  requirements 
analysis  process  in  place  to  address  cultural  training  requirements  and  that  cultural  skills  must  be 
placed  in  a  specific  mission  context.  The  group  also  focused  again  on  the  role  of  leaders,  how 
they  teach  and  model  respect,  and  how  they  reward  behavior. 

The  final  session  of  the  Analysis  track  focused  on  Cross-Cultural  Learning  as  a 
Developmental  Progression,  and  was  led  by  Dr.  Melissa  Brittain  from  the  Air  Force  Culture  and 
Language  Center.  Dr.  Brittain  began  her  briefing  by  noting  it  is  necessary  to  understand  how 
individuals  leam  cross-cultural  competence.  The  current  model  the  Air  Force  uses  focuses  on 
three  skills:  negotiation,  communication,  and  relating.  On  the  developmental  continuum,  they 
focus  on  consistent  traits,  emerging  traits,  key  learning,  and  transition  specific  critical 
experiences.  They  also  look  at  predisposition  to  think  about  putting  people  on  specific  tracks 
based  on  their  abilities.  It  is  also  important  to  look  at  environmental  considerations,  which  are 
all  unique  to  specific  Services.  Dr.  Brittain  described  the  Air  Force  rhythm  of  learning  on  a 
timeline  and  where  culture  education  fits  in.  Cultural  instruction  is  provided  in  education,  and, 
once  a  destination  is  known,  the  Air  Force  begins  to  provide  more  culture-specific  training. 

The  Air  Force  is  currently  conducting  competency  modeling  to  see  what  the  domain  of 
cultural  capability  looks  like.  They  are  conducting  interviews  in  theater  with  personnel  who 
have  taken  their  PMEs.  In  the  future,  the  Air  Force  hopes  to  have  a  model  that  is  operationally 
relevant  with  practical  cross-cultural  competence.  It  is  also  important  to  reinforce  long-term 
development  of  cross-cultural  competence  in  the  field  and  to  better  understand  the  role  of 
environmental  conditions.  Gaps  include  finding  what  skills  are  most  critical  for  certain 
populations,  what  role  individual  characteristics  play,  and  finding  ways  to  utilize  skills  in  the 
field.  Comments  included  the  need  to  show  some  consistency  within  training  and  the  need  to 
identify  right  training  for  right  person  for  right  location  at  right  time. 

The  second  half  of  this  session  was  presented  by  Mr.  Mike  McCloskey,  from  361 
Interactive,  and  focused  on  The  Developmental  Progression  of  Cross-Cultural  Competence  in 
Soldiers.  He  began  by  discussing  how  to  measure  and  assess  cross-cultural  competence,  noting  it 
is  important  to  look  into  burn-out  after  a  Soldier  has  been  deployed  for  an  extended  period  of 
time  and  their  continued  motivation.  361  Interactive  takes  a  multi-tiered  approach  to  assessing 
cultural  competence  within  Soldiers.  Mr.  McCloskey  described  team  ranking  task  interviews  that 
were  conducted  and  how  interviewees  described  themselves,  including  what  skills  more 
advanced  personnel  had.  In  addition,  they  conducted  critical  incident  interviews  to  find  likely 
causes  of  both  success  and  failures.  Simulation  interviews  were  used  to  probe  how  different 
aspects  of  cross-cultural  competence  would  develop,  how  expertise  develops  in  a  traditional 
domain,  and  how  to  integrate  that  with  culture.  Their  goal  is  to  develop  a  tool  to  assess  and 
provide  feedback  on  a  Soldier’s  level  of  cross-cultural  competence.  Mr.  McCloskey  discussed 
six  factors:  cultural  maturity,  empathy,  cognitive  flexibility,  awareness,  assessment  acuity,  and 
interpersonal  skills.  Mr.  McCloskey  then  described  his  developmental  model  of  C3  and  also  an 
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assessment  interview,  which  details  question  based  on  the  level  of  the  Soldier.  Experience  can  be 
instrumental  in  learning,  but  can  also  sometimes  be  detrimental  if  the  experience  is  negative  or 
consistently  stressful. 

Mr.  McCloskey  noted  the  importance  of  having  a  threshold  level  of  empathy,  and  how  it 
should  be  addressed,  including  foreign  interaction  and  basic  coaching.  Only  about  5%  of 
Soldiers  361  Interactive  interviewed  had  very  low  empathy  and  other  Soldiers  were  aware  of 
them.  Discussions  distinguished  empathy  from  sympathy,  and  some  participants  expressed 
concern  that  empathy  carries  risks.  Empathy  may  cause  Service  members  to  become  so 
relativistic  that  they  will  not  be  able  to  pull  the  trigger  when  needed. 

The  group  also  discussed  the  need  to  select  people  who  will  not  do  well  in  a  foreign  area 
and  reassign  them  to  other  fields.  Mr.  McCloskey  noted  that  in  the  corporate  realm,  in-country 
training  has  proved  highly  effective,  but  that  this  may  not  work  in  a  military  context.  The  group 
then  discussed  the  need  to  train  and  evaluate  Soldiers  in  cross-cultural  competence  since 
selection  is  not  an  option. 

Finally,  discussion  focused  on  experiences  in  the  field  during  deployment  and  the  need  to 
be  more  systematic  at  capturing  these  experiences.  There  is  a  database  being  built  at  Ft. 
Leavenworth  to  look  at  lessons  learned  in  this  area.  Ft.  Bragg  has  also  done  a  lot  of  work  with 
adaptability.  It  is  much  easier  to  train  hard  skills,  such  as  how  to  hold  a  gun,  than  soft  skills  like 
coping  with  cognitive  dissonance.  There  is  also  a  dilemma  in  creating  efficacy  without  having 
too  much  self-confidence. 

In  conclusion,  the  Analysis  track  looked  at  the  military  environment  and  leadership.  The 
Air  Force  representative  noted  they  have  a  difficult  time  providing  cultural  education  and 
training  once  they  move  up  in  the  military  ranks.  Global  businesses  face  the  same  issues.  Two 
options  were  brought  up  in  conclusion:  either  show  personnel  what  to  do  and  tell  them  to  do  it  or 
train  them  from  the  very  beginning. 


Analysis  Track  Recommendations 

Define  and  create  a  core  set  of  cross-cultural  competencies  for  the  military. 

The  military  needs  to  define  and  create  a  core  set  of  cross-cultural  competencies.  The 
core  set  should  include  a  foundational  doctrine  base  and  must  be  embedded  in  general  education 
and  training  using  a  multi-faceted  approach.  Once  a  core  set  has  been  defined,  the  military  must 
invest  in  understanding  precisely  how  individuals  learn  cross-cultural  competence,  as  well  as 
what  role  environmental  factors  play,  in  order  to  encourage  long-term  development. 
Additionally,  the  military  must  find  ways  to  both  measure  and  assess  cross-cultural  competence. 
The  need  for  a  core  set  of  cross-cultural  competencies  was  the  main  recurring  theme  throughout 
the  analysis  track.  The  group  also  recommended  reaching  out  to  a  broader  community,  beyond 
the  military,  to  get  input  in  how  to  achieve  the  this  recommendation. 
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Demonstrate  the  importance  of  culture  for  performance  outcomes  and  mission  success. 

Analysis  track  participants  emphasized  the  importance  of  teaching  culture  as  a 
fundamental  enabler  for  mission  performance.  The  need  for  Soldiers  to  be  competent  both  in 
conventional  skills  and  in  sociocultural  skills  was  recommended  in  more  than  one  session  and 
after  lengthy  discussion  from  participants.  Culture  must  be  taught  as  a  life-saving  skill,  including 
rudimentary  skills  in  interacting  with  foreigners,  prior  to  deployment.  The  military  cannot  rely 
on  Soldiers  to  receive  on-the-job  training  in  culture.  In  order  to  get  Service  members  motivated 
to  learn  about  culture  at  an  early  stage,  it  is  recommended  that  training  and  education  tie  cultural 
adaptability  and  agility  to  surviving  in  theater  early  on,  not  be  left  only  to  pre-deployment 
training.  It  was  noted  this  type  of  teaching  will  go  a  lot  further  than  simply  teaching  culture  as  a 
secondary  issue.  Simply  put,  culture  needs  to  become  a  value  within  the  military. 

Increase  cross-cultural  instructional  efforts  and  focus  on  the  curriculum. 

More  training  hours  must  be  added  for  culture;  however,  the  group  noted  that  this  is  a 
dilemma,  as  military  schools  and  training  programs  already  suffer  from  densely  packed  programs 
of  instruction  and  limited  time.  Analysis  track  participants  suggested  that  understanding  of  one’s 
own  cultural  and  language  is  an  important  pre-requisite  to  learning  about  a  foreign  culture  or 
language.  It  was  recommended  that  the  Services  invest  in  teaching  cultural  self-awareness  to 
junior  personnel  as  a  foundation  for  further  cultural  education  and  training.  Once  a  student  fully 
understands  his  own  culture  and  language,  he  will  have  an  easier  time  learning  about  another 
culture  and  learning  a  foreign  language.  It  is  important  to  find  a  way  to  separate  different 
requirements  for  different  job  descriptions  within  the  applicant  pool:  one  set  for  experts,  another 
for  non-experts.  Finally,  in  regards  to  training,  it  is  also  important  to  show  consistency  across 
the  field  and  to  identify  appropriate  training  that  caters  to  the  individual  at  the  correct  time. 


Design  Track  Summary 

Dr.  Michelle  Wisecarver  and  Mr.  Gonzalo  Ferro  (PDRI)  presented  the  first  session  of  the 
Design  track,  Designing  Cultural  Training  to  Maximize  Training  Transfer.  The  goal  of  cultural 
training  is  to  maximize  training  transfer,  but  how  does  one  enable  transfer  for  adaptive 
performance  in  which  the  tasks  and  conditions  are  changing?  Intercultural  adaptability  means 
producing  an  effective  response  to  environmental  change.  The  key  to  developing  instruction  for 
adaptability  is  to  address  metacognition,  to  create  retrieval  cues,  using  Merrill’s  five  principles  of 
instruction:  activation,  demonstration,  application,  integration,  and  task-centered  instruction. 
Discovery  and  error-based  learning  is  good  for  training  adaptability  because  trainees  can  see  how 
and  where  they  make  mistakes.  There  was  much  discussion  of  computer-based  training,  as  the 
presenters  introduced  software  developed  for  the  John  F.  Kennedy  Special  Warfare  Center  and 
School,  to  create  a  foundation  for  cultural  training  for  entry-level  SOF.  As  with  other  software 
packages  mentioned  by  other  members  of  the  group,  the  goal  is  to  develop  culture-general 
organizers  and  then  apply  this  to  culture-specific  knowledge.  Teaching  regional  facts  (such  as 
history  and  geography)  helps  build  a  foundation. 

There  was  also  great  debate  about  cultural  taxonomies  and  the  utility  of  a  taxonomy  of 
cultural  dimensions  to  drive  underpinnings  of  various  cultural  initiatives.  A  taxonomy  could  help 
organize  the  cultural  infonnation  to  be  conveyed  in  instruction,  or  help  trainees  to  organize  their 
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cultural  knowledge.  Salmoni  and  Holmes-Eber’s  dimensions  of  operational  culture  and 
Hofstede’s  dimensions  were  two  such  frameworks  discussed.  The  audience  agreed  that  there  was 
no  widely  accepted  taxonomy  to  date.  A  related  discussion  addressed  the  analogies  typically 
used  for  culture.  For  example,  some  felt  that  the  common  iceberg  analogy  is  insufficient  to 
convey  the  complexity  of  culture.  A  tree  analogy,  with  its  branches  and  roots,  might  be 
appropriate. 

The  group  also  focused  on  ways  to  motivate  warfighters  to  leam  about  culture.  The  time 
allocated  for  cultural  content  is  not  sufficient  to  teach  everything  the  Services  members  need  to 
know;  how  can  education  and  training  encourage  independent  and  lifelong  learning?  Participants 
discussed  warfighters’  tendencies  to  be  apathetic  until  they  get  to  theater,  and  then  want  to  leam 
everything.  One  method  is  to  get  to  them  to  learn  without  realizing  they  are  learning  and  embed 
the  infonnation  into  other  lessons.  It  is  important  to  emphasize  this,  which  includes  changing  the 
way  culture  is  taught.  While  top-level  leaders  in  Defense,  such  as  General  Petraeus,  General 
McChrystal,  and  Secretary  Gates  are  beginning  to  realize  the  importance  of  this,  it  must  trickle 
down  and  lead  to  organizational  changes. 

The  second  session,  Integrating  Culture-Specific  and  Culture-General  Objectives,  was 
led  by  Dr.  Richard  L.  Wolfel  from  the  U.S.  Military  Academy  (USMA).  He  is  the  chair  of 
intercultural  competence  at  the  Center  for  Languages,  Cultures  and  Regional  Studies.  USMA  is 
trying  to  create  second  lieutenants  who  have  the  culture-general  skills  to  succeed  in  their  military 
careers.  There  are  three  parts  of  cross-cultural  competence  taught  at  USMA:  knowledge,  skills, 
and  affect.  For  knowledge,  the  core  curriculum  and  goal  is  to  place  a  team  in  the  field  to  assess 
effectiveness,  professional  military  ethics,  and  the  historical  value  of  landscapes.  There  is  a 
power  relationship  behind  all  of  this.  Government  uses  culture  to  promote  its  goals.  For  skills, 
the  goal  is  to  understand  general  regional  concepts  (not  region-specific  facts)  and  to  develop 
confidence  through  cross-cultural  communication.  Cadets  can  think  “I  did  this  and  survived — I 
can  do  this  again.”  This  helps  prevent  reverse  culture  shock  upon  return. 

USMA  offers  a  study  abroad  event  for  ten  days  during  spring  break,  and  a  summer 
program  for  21  days.  Additionally,  they  provide  semester  abroad  opportunities  that  give  Cadets 
the  opportunity  to  learn  a  language  through  immersion.  Currently,  USMA  can  send 
approximately  1,000  Cadets  abroad  at  once,  with  six  to  ten  Cadets  per  trip.  There  are  over  50 
countries  available  to  travel  to,  which  are  based  on  the  strategic  language  list.  This  complements 
their  knowledge  and  helps  build  skills  and  self-confidence  for  working  with  foreign  cultures.  Dr. 
Wolfel  then  described  how  this  is  applied  beyond  USMA.  All  Cadets  get  pre-  and  post-  training 
and  the  returning  Cadets  brief  the  outgoing  Cadets.  The  goal  at  USMA  is  to  develop  Soldiers 
who  will  feel  comfortable  interacting  with  other  cultures,  regardless  of  the  specific  culture.  Dr. 
Wolfel  has  used  the  Intercultural  Development  Inventory  and  a  Scenario  Based  Assessment  tool 
in  order  to  gauge  the  Cadets’  cultural  mindset  (from  monocultural  to  multicultural). 

The  co-leader  of  this  session  was  Dr.  Mark  Mendenhall  (University  of  Tennessee- 
Chattanooga).  The  business  world  has  provided  many  valuable  lessons  on  how  cultural 
adaptability  affects  job  perfonnance.  He  describe  a  cycle  consisting  of  the  ability  to 
perceive/analyze/decode  what  is  going  on,  to  accurately  identify  appropriate  action  (kinetic/non- 
kinetic),  to  possess  the  behavioral  flexibility  and  discipline  to  act  appropriately,  then  to 
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perceive/analyze/decode  again,  which  allows  people  to  function  effectively  in  foreign  cultures. 
However,  there  are  missing  links  in  most  cultural  training  and  education.  The  missing  link 
between  culture-general  and  cultural-specific  models  is  a  cultural-specific  manifestation  of  the 
general  models.  This  can  help  people  understand  what  to  do  in  culture-specific  situations. 
Soldiers  need  to  understand  the  “why”  behind  cultural  mechanics  to  bridge  the  gap  between 
knowledge  and  application.  There  is  always  a  system  that  undergirds  the  behaviors;  only 
understanding  the  framework  does  not  work.  This  system  provides  a  powerful  tool  to  understand 
what  to  do  in  any  given  situation.  It  is  necessary  to  include  this  missing  link,  which  occurs 
between  perceive/analyze/decode  and  accurately  identifies  concepts  within  the  cycle. 

Some  discussion  focused  on  individual  personality  traits  and  competencies,  such  as 
emotion  regulation,  and  how  these  affect  cultural  integration  strategies  and  how  individuals 
compensate  for  weak  traits  with  their  strong  traits.  Personality  traits  are  not  usually  malleable  so 
it  is  important  to  learn  how  to  utilize  them.  Further,  it  is  important  to  teach  that  understanding 
foreigners  does  not  mean  being  wann  and  soft,  it  means  being  able  to  connect  to  them  when 
needed  to  get  the  job  done. 

During  the  discussion  session,  Dr.  Mendenhall  stated  that  pre-deployment  intercultural 
skills  did  improve  intercultural  effectiveness,  but  that  the  most  important  thing  was  the  ability  to 
listen  and  the  sincerity  in  people  skills.  The  three  dimensions  most  related  to  intercultural 
effectiveness  are  cosmopolitanism,  other  orientation  (interest  and  relationship  engagement),  and 
self-management.  Emotional  resilience  and  optimism  may  also  play  a  role.  The  Myers-Briggs 
Scale  was  mentioned  during  the  question-and-answer  period,  but  this  scale  has  not  been  applied 
to  the  intercultural  domain  and  is  not  used  in  the  literature. 

The  third  session,  Gaming  and  Simulation  for  Intercultural  Communication,  was  led  by 
Dr.  Elaine  Raybourn  (Sandia  National  Laboratories)  and  Dr.  Mike  van  Lent  (Soar  Technology, 
Inc).  Dr.  Raybourn  discussed  her  experience  developing  intercultural  simulations.  These 
simulations  are  not  stand-alone;  the  instructor  is  heavily  involved.  Simulations  create  practice 
environments,  not  training,  which  is  complementary  to  live  action  training.  The  goal  of 
simulation  is  to  execute,  through  communication,  one’s  adaptiveness,  and  to  have  the  ability  to 
expertly  change  one’s  behavior  according  to  multicultural  situations.  What  must  still  be 
addressed  is  how  to  create  environments  that  allow  people  to  exercise  their  adaptiveness  and 
understand  their  situation.  Humans  in  the  AI  environment  are  role-played  by  other  trainees 
(learning  how  to  negotiate  and  play  both  sides)  or  experts.  They  draw  from  experiential  learning, 
metacognition,  and  adaptive  training.  Social-process  simulations  are  intended  to  help  people 
realize  that  assumptions  are  not  always  the  best  guide  for  action  and  to  allow  participants  to 
experience  what  it  is  like  to  behave  adaptively.  They  start  with  a  particular  challenge,  they  go 
through  actions  to  question  their  own  beliefs  which  create  a  conflict,  and  they  use 
communication  to  solve  the  problem.  Sometimes  the  realization  comes  during  after-action 
review.  The  focus  is  on  learning,  affect,  and  believability.  The  speech  acts  are  parsed  out  into  a 
narrative.  In-game  feedback  becomes  after-action  review  and  trainees  often  come  out  of  the 
experience  with  a  broader  solution  space. 

Dr.  van  Lent  discussed  modeling  a  culturally-specific  individual  for  use  in  computer- 
based  simulation  and  training.  He  described  how  visuals,  such  as  appearance  and  gestures,  are 
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just  the  tip  of  the  iceberg.  A  greater  focus  is  the  beliefs,  goals,  schemas,  norms,  emotions,  and 
personality  that  drive  behavior,  rather  than  scripting.  Furthermore,  instead  of  hiring  a  lot  of 
people  to  do  role-playing,  he  uses  culturally-specific  virtual  role-players.  Designing  the  training 
or  simulation  requires  a  set  of  models  that  include  what  is  going  on  with  the  student  and  what  the 
student  should  know.  These  models  include  an  expert  model  (an  ideal  that  serves  as  the  target), 
an  assessment  model  to  measure  the  learner’s  progress,  a  model  of  the  learner,  a  pattern  model  to 
allow  for  comparison  between  the  learner  and  a  referent  group  of  peers.  A  monitor  checks  what 
is  happening  and  gives  tailored  feedback,  which  can  be  in  the  form  of  explicit  feedback  (telling 
them  what  they  did  wrong)  or  implicit  feedback  (adapting  scenario  based  on  what  they  do).  Dr. 
van  Lent  discussed  the  use  of  these  models  in  a  project  currently  in  development,  Social  Cultural 
Immersive  Learning  (SCIL),  which  teaches  culture-general  skills  that  can  help  a  student  in  any 
scenario. 

Much  discussion  focused  on  the  definitions  of  and  distinctions  between  games  and 
simulation.  There  is  no  technological  distinction  between  games  and  simulations  except  for  the 
game  engine.  Differences  are  primarily  a  matter  of  emphasis.  Games  generally  direct  a  user’s 
experience  down  a  path,  and  simulation  provides  them  with  more  freedom.  A  typical  game  puts 
more  emphasis  on  engagement  and  a  story.  Games  depend  on  believability,  not  realism.  Dr.  van 
Lent  discussed  the  importance  of  knowing  how  real  a  game  needs  to  be  for  positive  training 
impact  and  the  tradeoff  of  virtual  vs.  live-action  films  (facial  expressions)  vs.  live  role  players 
(creativity)  to  occur.  Immersion  can  have  benefits,  but  may  also  have  drawbacks.  Immersion  can 
affect  a  student’s  emotional  response  and  make  the  student  feel  stressed.  It  is  a  weak  form  of 
emotion,  but  being  caught  up  in  the  game  can  help  in  the  real  event.  However,  it  is  not  known  if 
being  caught  up  in  the  immersion  helps  one  to  retain  knowledge.  In  any  case,  either  games  or 
simulation  can  be  a  fruitless  endeavor  if  not  properly  instantiated  to  support  a  training  objective 
and  become  a  training  objective. 

Another  area  of  controversy  is  the  role  of  fun,  and  whether  it  causes  the  student  to  spend 
more  time  in  the  immersion  or  if  content  is  more  important.  Fun  does  not  equate  to  being 
engaging,  stimulating,  or  motivating,  which  are  the  important  aspects  of  a  simulation.  One  of  the 
aims  is  to  get  learners  to  use  the  knowledge  learned  outside  of  the  classroom  because  they  enjoy 
it,  hopefully  working  towards  learning  objectives,  so  fun  can  be  a  motivational  factor.  In 
addition,  learners  with  video  game  experience  will  bring  that  prior  experience  and  the 
accompanying  expectations  to  the  classroom.  However,  the  important  point  is  that  fun  can  be  a 
learning  tool,  but  the  military  is  not  under  a  moral  responsibility  to  make  things  fun  for  the  sake 
of  being  fun.  It  is  more  important  to  do  the  best  thing  to  achieve  the  training  outcome.  Whether 
having  fun  leads  to  better  learning  requires  research. 

Effective  use  of  stories  in  instructional  design  is  another  area  for  future  research.  Stories 
are  an  efficient  way  to  communicate  tacit  knowledge  and  military  leadership  development. 
Human  brains  are  wired  and  better  designed  to  accept  stories  than  information  in  raw  form. 

Extensive  discussion  focused  on  believability  and  physical  fidelity  requirements. 
Participants  discussed  whether  the  fidelity  of  a  simulation  is  equivalent  to  credibility  and  how 
lifelike  avatars  need  to  be.  The  simulations  do  not  need  to  be  hyper-realistic  in  terms  of  details, 
but  believable  that  characters  are  emotional  beings.  Physical  fidelity  in  this  domain  is  not  the 
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right  question;  rather  the  right  question  is  what  the  cognitive  model  fidelity  is.  For  example, 
animated  movies,  such  as  Toy  Story  or  A  Bug’s  Life,  use  unrealistic  characters,  but  are  highly 
successful  as  engaging,  living  characters.  Cognitive  task  analysis  will  define  cognitive  model 
fidelity  requirements.  Simulation  development  should  put  resources  into  that  instead  of  details 
such  as  blades  of  grass.  Some  medical  simulations  have  no  graphics,  only  text  which  means 
relying  on  symptoms  to  yield  a  diagnosis.  It  is  problem-based  education,  engages  the  student, 
and  has  to  deal  with  mental  representations.  It  is  the  opposite  of  believability,  but  very 
successful.  It  is  important  to  look  at  how  much  the  trainee  feels  the  system  is  helping  him  leam. 

Service  members  seem  to  have  very  high  expectations  of  fidelity;  they  expect  to  see  a 
video  example  over  3D/2D  avatars.  But  even  for  video,  they  reject  the  authenticity  in  them  and 
have  tended  to  find  small  discrepancies.  Participants  discussed  whether  an  investment  in  greater 
physical  fidelity  would  pay  off  in  terms  of  greater  learning  or  acceptance  from  warfighters.  The 
up-front  motivation  of  cultural  novices  was  addressed.  There  is  a  challenge  of  connecting  with 
someone  who  knows  an  operational  scenario  versus  someone  who  is  unfamiliar  with  the 
situation.  To  aid  in  this,  the  situation  must  be  described  so  that  it  becomes  personally  relevant. 
One  person  thought  having  senior  leadership  explain  why  a  situation  is  important  could  be 
helpftd. 


The  choices  available  and  resulting  feedback  in  a  simulation  or  game  were  viewed  to 
have  an  important  impact  on  believability.  The  challenge  is  to  build  back-ends  that  support  lots 
of  situations  and  decision-making  processes.  It  is  necessary  to  find  the  sweet  spot  in  terms  of 
how  many  choices  and  how  much  feedback  is  given.  A  choice-feedback-choice  chain  breaks  the 
realism.  Instead,  one  could  focus  on  a  step-by-step  multi-choice  chain  mapped  into  a  3D 
environment.  Choice-choice-choice-choice-feedback  or  subtle  feedback  are  other  possibilities, 
but  whether  one  must  stop  the  scenario  after  each  decision  to  give  feedback,  was  debated  by  the 
group.  They  also  discussed  what  the  level  of  granularity  of  decisions  that  maximizes  one’s 
feedback  and  training  should  be  and  how  much  learning  is  transferred  during  multiple  choices. 
The  group  discussed  the  importance  of  aiming  to  target  higher-level  skills  such  as  metacognitive 
prompts.  This  can  be  done  by  asking  the  user  how  he  made  a  decision  or  how  he  combined 
knowledge  and  information  and  whether  he  included  intervention  between  scenarios.  This  could 
be  accomplished  in  an  after-action  review. 

The  fourth  session  in  the  Design  Track  was  led  by  Mr.  Doug  Nelson  (Kinection)  and  Dr. 
David  Matsumoto  (San  Francisco  State  University),  Translating  Intercultural  Competencies  into 
Learning  Objectives.  They  described  competencies  as  a  set  of  behaviors  that  describes  excellent 
performance.  Dr.  Matsumoto  discussed  how  to  take  big  picture  competencies  and  break  them 
down,  noting  not  to  worry  about  how  they  were  obtained,  but  rather  what  the  problems  are  in  the 
real  world  and  which  problems  they  want  to  solve.  It  does  not  describe  a  particular  behavior,  so 
different  people  could  solve  it  by  focusing  on  different  behaviors.  Academics  bring 
competencies  from  literature,  but  no  one  does  the  problem  extraction  from  the  field.  The 
challenge  in  the  cross-cultural  area  is  that  there  is  no  defined  list  of  competencies;  it  is  needed  to 
recreate  a  competency  model.  Such  a  model  might  include  the  basics  to  engage,  communicate, 
read  situations,  build  relationships,  and  negotiate,  with  the  goal  as  having  a  cross-culturally 
competent  warfighter.  The  presenters  then  gave  the  group  a  list  of  KSAs  to  use  for  breakout 
groups.  The  groups  were  told  to  filter  this  list  against  the  competencies  found  in  the  breakout 
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groups  and  see  if  it  corresponds.  There  was  a  consensus  that  having  general  concepts  is 
beneficial. 

Data  from  the  field  suggest  that  knowledge  gains  do  not  necessarily  translate  to 
performance  gains.  In  terms  of  knowledge,  learning  objectives  are  the  specification  of  what  a 
student  should  leam.  These  should  be  focused  on  the  learner,  not  the  teacher,  and  what  he  will  be 
able  to  know,  feel,  or  do  differently  as  the  result  of  an  intervention.  These  objectives  should  be 
demonstrable  and  assessable,  and  one  should  be  able  to  rate  the  success.  This  is  important 
because  without  good  goals,  one  cannot  assess  progress  or  performance.  Specific,  concrete 
learning  objectives  should  drive  instructional  design  and  will  help  the  community  get  beyond  the 
knowledge  domain  to  application. 

The  difference  between  knowledge,  skills,  and  attitudes  are  that  knowledge  is  easy  to 
teach  and  assess  and  includes  things  one  can  talk  about,  whereas  skills  are  things  one  can  do,  and 
attitudes  are  things  that  one  feels.  The  group  discussed  how  much  of  this  skill-building  is 
because  of  cross-cultural  skill  differences.  Some  of  these  are  cultural  rules  that  breakdown  to 
create  knowledge,  but  the  difference  between  knowledge  and  skills  is  the  ability  to  act  properly 
versus  what  one  knows.  Experiential  exercises  may  hit  some  of  the  skills  and  more  time  should 
be  spent  planning  the  exercise  to  hit  multiple  points.  Knowledge-based  assessment  works  if  the 
instructor  explains  why  (“states  the  reasons”)  but  never  asks  the  trainees  to  demonstrate  the  task 
(which  would  be  a  skill). 

The  track  concluded  its  final  session  by  breaking  into  smaller  groups.  Armed  with  a  list 
of  KSAs,  the  goal  was  to  design  and  develop  training  for  USMA  cadets  in  KSAs  to  develop 
tolerance  for  ambiguity.  Discussions  noted  the  difficulty  of  assessing  affective  learning 
objectives  and  of  establishing  the  reliability  and  validity  of  measures  selected  to  assess  learning. 


Design  Track  Recommendations 

Instructional  interventions  should  address  motivation  as  a  learning  outcome. 

Motivation  for  cultural  learning,  self-confidence  in  one’s  ability  to  learn  and  apply 
cultural  skills,  and  valuing  the  role  of  cultural  knowledge  and  skills  in  supporting  the  mission 
were  all  motivational/affective  learning  outcomes  that  discussions  cited  as  important.  Research  is 
needed  to  detennine  what  methods  best  achieve  these  outcomes.  Using  stories  and  scenarios  with 
personal  relevance,  highly  immersive  simulations,  and  messages  from  senior  leaders  were 
proposed  as  effective  ways.  It  is  important  to  assess  the  impact  of  these  methods  on  affective  and 
motivational  learning  outcomes. 

Conduct  research  on  the  role  of fun,  fidelity,  and  feedback  in  synthetic  learning  environments. 

These  three  issues  were  the  topic  of  much  discussion  A  better  understanding  of  their 
contributions  to  cultural  learning  would  enable  better  design  of  games  and  simulations  to  support 
learning  and  adaptive  transfer.  Participants  suggested  more  research  be  conducted  into  which 
types  of  feedback  are  most  effective,  what  level  of  granularity  for  decisions  and  feedback,  how 
fun  and  enjoyment  affect  learning  and  motivation,  and  what  level  of  fidelity  is  needed  to 
maximize  learning. 
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Develop  methods  and  metrics  for  assessing  learning  and  learning  transfer. 

There  are  currently  no  accepted  metrics  to  assess  learning  or  to  evaluate  training  for 
culture,  particularly  for  affective  and  metacognitive  learning.  Behavioral  performance  can  be 
observed,  but  some  learning  outcomes  that  are  important  in  enabling  effective  performance 
cannot  be  directly  observed  and  are  therefore  more  difficult  to  assess.  Assessment  methods  are 
needed  to  support  instructional  design,  training  evaluation,  and  linkages  between  learning  and 
performance.  Assessment  would  also  allow  for  a  detennination  of  whether  cultural  learning  has 
any  negative  impact  on  more  conventional  skill  sets. 


Development  Track  Summary 

The  first  session  of  the  Development  track  began  with  a  briefing  by  Dr.  Robert  Sands 
(Air  Force  Culture  and  Language  Center)  entitled,  Using  Narrative  and  Ethnographic  Methods 
in  Cultural  Instruction.  There  are  different  ways  to  use  ethnography  to  educate  from  a 
pedagogical  standpoint.  Ethnographic  methods  can  be  used  to  elicit  narratives  for  instruction. 
Narrative  can  be  defined  as  a  primary  form  of  thought  that  has  evolved  to  serve  functions  of  1) 
communicating,  2)  a  repository  for  culture-  specific  and  general  social  knowledge  expressed 
through  myth  and  story,  3)  as  a  means  to  provide  an  effective  platform  to  prepare  for  future 
activities.  The  Air  Force  uses  narrative  as  both  a  resource  and  a  means  of  assessment.  Dr.  Sands 
is  collecting  empirical  data  using  Airmen  narrative,  in  which  Airmen  discuss  their  stories  of 
deployment,  without  the  use  of  interview  questions  to  guide  them.  Some  of  these  stories  are 
recorded  on  video  and  then  used  in  culture -related  courses.  These  narratives,  which  bring  more 
intensity  and  emotion  into  the  experience,  assist  in  training  future  Airmen  what  to  expect  when 
they  are  deployed  in  the  field. 

Ms.  LeeEllen  Friedland  presented  narrative  methods  as  used  in  Alelo’s  Operational 
Language  and  Culture  Training  System  (OLCTS)  and  the  Virtual  Cultural  Awareness  Trainer 
(VCAT),  as  well  as  in  virtual  reality  (VR)  training  systems.  Alelo  creates  serious  games  and 
provides  a  mechanism  for  training  inter-communicational  skills.  The  goals  of  OLCTS  are  to 
develop  a  training  environment  to  acquire,  rehearse  and  apply  relevant  language  and  culture 
skills  in  a  mission.  The  VCAT  is  web-based  and  can  be  delivered  on  a  handheld  device.  Alelo 
uses  an  applied  ethnographic  approach  to  develop  elements  of  curriculum  design  and  course 
content.  In  particular,  Alelo  developed  a  situated  culture  methodology  that  focuses  solely  on 
things  necessary  for  mission  success.  Alelo  also  uses  ethnographic  interviews  and  role-playing 
with  Subject  Matter  Experts  to  assist  in  developing  dialogue  which  can  later  be  used  in  scenario 
practice.  Both  the  data  and  narratives  are  shaped  by  research  design  and  methods.  Various  types 
of  narratives  are  collected,  in  which  Subject  Matter  Experts  are  asked  specific  questions  as  well 
as  to  describe  specific  things  and  to  discuss  their  personal  experiences. 

Discussion  focused  on  bias  which  occurs  when  warfighters  are  provided  a  post¬ 
deployment  narrative,  especially  since  opinions  can  differ  so  much  between  individuals  and 
situations.  To  mitigate  this  effect,  Alelo  takes  many  samples  in  order  to  triangulate  and  find 
commonalities  among  the  narratives.  Other  concerns  included  how  much  of  this  type  of  training 
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Airmen  actually  retain,  and  it  was  noted  that  the  use  of  hand-held  devices  allow  Airmen  to 
refresh  their  training.  Because  of  the  narrative  involved,  the  learner  feels  more  connected  to  what 
they  are  learning.  Another  issue  that  was  raised  was  the  risk  of  warfighters  becoming  less 
adaptable  to  situations  and  instead  simply  memorizing  how  to  respond.  To  remedy  this,  Alelo 
exposes  its  students  to  multiple  scenarios  and  includes  role-playing  in  its  training,  which  differs 
from  the  narratives  they  watch,  in  order  to  ensure  they  learn  to  think  quickly  and  independently. 

The  Development  group  discussed  the  importance  of  Soldiers  having  a  realistic  view  of 
what  a  foreign  area  looks  like,  so  that  regardless  of  what  it  looks  like,  they  treat  it  with  respect. 
They  also  noted  the  need  for  the  narrators  to  have  a  high  level  of  empathy  in  order  to  get  the 
right  thing  across  to  the  trainees,  and  also  for  the  narrators  to  discuss  perceptions  of  Americans 
and  attitudes  toward  Westerners  in  foreign  countries,  to  best  prepare  warfighters  for  what  to 
expect  when  deployed.  Thus,  it  is  important  to  show  a  proper  balance  of  both  positive  and 
negative  experiences  for  students  to  view  and  understand. 

The  group  also  discussed  cultural  training’s  target  audience,  noting  it  is  variable  and 
constantly  changing.  Some  people  thought  that  it  is  not  good  to  give  a  lot  of  cultural  training 
pre-deployment,  as  it  is  a  small  priority,  when  Soldiers  are  heavily  focused  on  whether  their 
annor  fits.  It  is  important;  therefore,  to  focus  on  cultural  training  at  appropriate  times,  although, 
timing  and  technology  remain  constraints  in  reaching  the  target  audience.  It  is  also  imperative 
that  training  include  both  general  as  well  as  specific  cultural  knowledge,  and  also  that  it  be 
designed  for  the  target  audience,  such  as  using  Marine  videos  to  train  Marines  and  not  Airmen, 
so  that  each  Service  can  best  relate. 

It  is  also  important  to  teach  culture  as  a  life-saving  skill,  and  to  first  assess  where  a 
Soldier  is  in  their  own  cultural  development  in  order  to  best  customize  training  to  the  individual. 
It  was  noted  that  Marines  who  found  out  they  were  successful  in  their  mission  were  more  eager 
to  discuss  their  experiences.  However,  some  people  noted  that  the  word  empathy  should  not  be 
taught  to  Marines,  as  it  distracts  them  from  their  main  mission  focus  and  that  they  will  not  take 
cultural  empathy  seriously.  Furthennore,  it  was  of  opinion  that  cultural  empathy  is  not  how 
Marines  think  of  themselves  and  it  should  therefore  not  be  taught  at  boot  camp.  This  led  to 
further  discussion  on  the  difference  between  cultures  in  the  different  branches  of  the  military, 
and  it  was  repeated  that  each  Service  must  instruct  its  personnel  in  a  way  that  correlates  with  its 
own  culture.  An  across-the-board  training  program  will  not  work. 

The  session  concluded  by  discussing  the  challenge  of  storing  data.  A  database  for 
cultural  data  and  narratives  could  allow  the  entire  Department  of  Defense  to  access  it  and  use  it 
to  build  accurate  models  and  instructional  materials. 

The  second  session,  Cognitive  Load  and  Other  Considerations  for  Multimedia  Delivery, 
began  with  a  briefing  by  Mr.  Gary  Rauchfuss  (U.S.  Army  Training  and  Doctrine  Command, 
NCO  Development).  Mr.  Rauchfuss  discussed  moving  the  Army  from  an  outdated  training 
design  which  involves  subject  matter  experts  developing  training,  to  a  more  modern  approach, 
which  uses  the  latest  findings  in  training  research  and  applies  it  to  military  training.  Mr. 
Rauchfuss  focused  specifically  on  research  on  cognitive  load  theory  (e.g.,  van  Merrienboer  & 


17 


Sweller,  2005).  It  is  important  to  understand  how  much  intrinsic  load  the  subject  matter  has;  in 
cultural  training,  this  is  likely  a  significant  amount.  It  is  then  important  to  balance  the  type  of 
content  being  taught  with  the  format  in  which  it  is  administered.  In  order  to  prevent  the  content 
from  overwhelming  the  learning,  and  to  reduce  an  extraneous  load  of  material,  the  content  needs 
to  be  broken  down  into  smaller  concepts  to  make  them  easier  to  learn.  For  example,  people  learn 
better  from  visual  and  audio  aids  than  visual  and  written  aids.  It  is  therefore  important  to  use 
modalities  effectively. 

It  is  also  important  to  understand  definitions  in  order  to  understand  bigger  concepts. 
Therefore,  one  should  teach  definitions  first,  rather  than  embedding  definitions  in  other  concepts. 
Smaller  definitions  and  concepts  should  be  presented  before  the  bigger  concepts,  so  that  the  load 
is  not  too  big  to  try  to  have  the  student  learn  small  concepts  embedded  along  the  way. 
Additionally,  it  is  recommended  that  speech  accents  of  trainers  be  identical  to  accents  of 
learners,  and  that  trainers  personalize  their  teaching  in  the  second  person  (e.g.,  use  “you  will  be 
learning”  instead  of  “the  student  will  be  learning”).  It  is  also  important  to  reduce  load  by  using 
spatial  contiguity;  i.e.  putting  labels  on  diagrams,  and  keeping  text  and  graphics  on  the  same 
pages.  Finally,  Mr.  Rauchfuss  discussed  using  temporal  contiguity,  such  as  having  audio  paired 
with  animation,  so  as  not  to  impose  additional,  unnecessary  load  on  the  learner. 

The  second  half  of  this  session  was  led  by  Dr.  Chad  Lane,  who  discussed  intelligent 
tutoring  systems  and  feedback.  He  also  discussed  effective  ways  to  provide  pedagogical  support 
in  immersive  learning  environments.  One  example  of  this  is  using  transfer  and  retention  to  create 
robust  learning.  Formative  feedback  should  be  non-evaluative,  supportive,  timely,  and  specific 
(Shute,  2008).  Instead  of  having  the  learner  memorize  a  goal,  it  is  visually  on  a  screen.  It  is 
important  to  consider  whether  people  will  even  pay  attention  to  things  like  a  tutoring  window. 

To  test  this,  one  can  use  eye  tracking  to  see  where  people  actually  look.  Better  tutors  force 
learners  to  use  knowledge  recently  gained  to  answer  questions  and  consider  issues.  The  goal  of 
this  is  to  get  the  learner  to  think  more  and  to  retain  the  learned  information.  It  is  also  important  to 
balance  the  tension  between  using  a  convincing  story  and  teaching  a  concept.  UrbanSim,  for 
example,  is  a  game-based,  interactive  way  to  learn  about  counterinsurgency  in  any  foreign 
theater.  Different  models  have  different  simulation  systems.  BiLAT,  for  example,  is  driven  by 
timing,  appropriateness,  and  relevance  of  actions,  whereas  UrbanSim  is  driven  by  the  closeness 
to  the  ideal  end  state. 

The  Development  Track  then  held  an  open  dialogue  and  had  a  question- and-answer 
session  on  the  presentations.  One  person  thought  UrbanSim  might  be  too  much  of  a  cognitive 
load,  since  students  are  given  binary  questions  to  answer.  This  is  potentially  an  issue  depending 
what  the  learner  is  thinking  at  the  time.  A  designer  in  the  group  noted  designers  have  yet  been 
able  to  apply  knowledge  on  learning  and  teaching  to  create  effective  games.  If  the  trainer  can  see 
what  the  learner  is  doing  during  the  simulation,  then  it  can  be  assessed,  to  know  the  student  will 
practice  what  they  should  be  learning.  Otherwise,  it  is  essential  for  the  learner  to  be  engaging  in 
further  instruction.  The  system  is  designed  for  the  learner  to  both  practice  what  they  just  learned 
as  well  as  to  learn  a  new  concept.  When  asked  what  issues  might  affect  a  learner’s  cognitive  load 
using  the  UrbanSim  program,  answers  included  the  number  of  icons  and  amount  of  text.  To 
mitigate  this,  students  must  be  pre-trained  on  the  game  and  how  to  interact  with  it.  To  avoid 
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stress,  players  are  not  given  a  specific  time  limit  to  accomplish  tasks.  The  group  also  discussed 
how  effective  this  type  of  virtual  environment  is  in  creating  emotional  responses,  and  what  level 
of  fidelity  is  necessary.  The  group  also  discussed  where  to  invest  in  cultural  training,  but  did  not 
come  to  any  conclusions. 

The  group  also  discussed  multimedia  as  a  training  tool  in  general,  noting  that  future 
learners  will  be  more  familiar  with  multimedia  and  are  able  to  multitask  better  than  previous 
generations.  It  was  noted  that  much  anecdotal  evidence  exists  in  this  realm,  but  there  is  no 
empirical  evidence.  In  fact,  research  has  shown  that  although  younger  generations  are  easily  able 
to  use  multimedia,  they  are  not  actually  more  effective  in  learning  through  simulation  than  older 
generations.  The  group  also  discussed  ways  to  give  learners  the  tools  to  promote  and  develop 
metacognitive  skills  and  the  role  of  instructors  in  facilitating  learning  in  these  synthetic  learning 
environments. 

The  third  session,  Theoretical  Foundations  for  Developing  Virtual  Humans  for  Training 
and  Simulation J  began  with  a  presentation  from  Ms.  Julia  Kim  (USC  Institute  for  Creative 
Technologies).  Ms.  Kim  discussed  how  to  best  design  virtual  humans  and  their  cognitive, 
emotional  systems.  She  is  interested  in  how  to  pick  and  choose  how  to  develop  characters  in 
virtual  environments.  Virtual  humans  are  able  to  enable  interpersonal  human  to  human 
communication  beyond  just  practicing  culture  on  a  macro  scale.  Additionally,  negotiation, 
language  skills,  and  general  interpersonal  communication  are  all  elements  that  can  be  explored 
using  virtual  humans.  Since  culture  is  embedded  in  everything  people  do,  and  how  they 
communicate,  simulations  allow  culture  to  be  taught  using  avatars.  Serious  games  have  come  a 
long  way  in  terms  of  more  realistic  looking  graphics  and  realistic  humans.  Culture,  which  plays  a 
role  in  perception  and  language,  can  be  shown  with  avatars  using  non-verbal  behavior.  Each 
avatar  is  an  autonomous,  intelligent  agent  that  acts  on  a  theory  of  emotion  that  is  programmed. 
Different  actions  of  the  learner  change  levels  of  trust  in  the  avatar  which  influence  how  the 
avatar  acts.  The  model  presented  can  change  on  the  fly,  based  on  inputs  that  the  user  also  has 
access  to. 

In  the  culturally  affected  behavior  (CAB)  system,  values  and  norms  are  overlaid  on  the 
task/plan  models.  Elements  such  as  how  avatars  interact  with  one  another  all  have  cultural 
elements  that  are  meant  to  be  instructional.  It  is  also  important  to  have  a  training  element  that 
explains  why  it  is  that  an  avatar  responds  in  a  certain  way.  At  ICT,  they  are  trying  to  explore 
means  for  making  simulation  more  educational.  This  can  include  having  an  avatar  explain  why  it 
is  that  he/she  reacted  a  certain  way,  or  whether  something  else  pops  up  and  informs  the  learner 
why  something  happened  the  way  it  did.  However,  it  is  a  big  challenge  to  figure  out  how  to  make 
the  models  apply  to  various  cultures.  Many  things  still  need  to  be  looked  at:  time  as  a  factor, 
knowing  what  to  focus  on,  how  interactions  should  be  driven  -  by  cognitive  or  behavior  models, 
and  assessing  how  to  ensure  users  are  learning  the  correct  lessons. 

The  next  part  of  this  session  was  presented  by  Dr.  Lynn  Smith-Lovin  (Duke  University). 
She  began  by  noting  that  in  order  to  develop  a  virtual  human,  one  has  to  build  on  a  model  that 
shows  how  humans  act  and  what  they  do.  Dr.  Smith-Lovin  is  developing  a  behavioral  model  for 
how  humans  respond  emotionally,  how  they  think,  and  what  attitudes  they  have,  all  of  which  are 
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necessary  for  programming  virtual  humans.  She  discussed  the  Affect  Control  Theory,  a 
mathematical  theory  that  has  the  precision  most  programmers  can  make  use  of.  Other  models 
are  in  verbal  terms  and  are  therefore  not  possible  to  program,  whereas  this  model  is  explicitly 
about  how  to  use  cultural  tools  within  a  culture  to  respond  to  others  and  to  respond  emotionally 
to  real  world  events.  Human  Terrain  states  that  every  block  of  a  village  differs,  which  is  not 
helpful  in  planning  operations  or  creating  virtual  trainers,  because  agent-based  models  assume  all 
humans  act  the  same.  This  model  allows  a  person  to  generate  understanding  about  an  infinite 
number  of  social  events  and  interactions.  Dr.  Smith-Lovin  noted  that  simulation  allows  the 
choosing  of  which  culture  to  use,  interact  with,  and  where  the  simulation  will  take  place.  It  can 
also  assign  actor  emotions.  This  model  allows  for  both  a  general  cultural  approach  as  well  as  a 
specific  knowledge  approach  and  is  feasible  for  putting  on  an  iPod  or  other  handheld  device  for 
warfighters  to  train  with. 

The  Development  group  discussed  the  potential  in  this  simulation  program  to  create  an 
element  of  surprise.  However,  there  has  not  been  a  lot  of  validation  of  the  cross-cultural  aspects 
of  this  model.  Data  is  collected  in  a  lab,  and  then  the  model  is  tested  in  environments.  The  group 
was  somewhat  confused  as  to  how  this  model  works  and  is  validated,  so  a  demonstration  was 
given.  Dr.  Smith-Lovin  also  noted  that  every  set  of  actions  is  contingent  on  the  previous 
interaction  history,  and  it  builds  up  with  every  event. 

To  clarify  between  the  two  presentations  during  this  session,  it  was  explained  that  the 
first  model  must  state  what  it  means  for  example,  to  assail  someone,  and  would  need  to  include  a 
set  of  verbal  or  visual  emotions  to  correlate  with  that.  Given  all  the  interactions  of  the  particular 
person,  one  can  see  why  the  individual  would  behave  the  way  it  does  and  what  the  consequences 
for  the  set  of  actions  would  be  using  different  kinds  of  people.  This  model  has  two  approaches: 
one  is  bottom  up  and  begins  with  the  model,  the  other  is  top  down.  Ideally  the  two  models  would 
meet  somewhere  in  the  middle.  Integrating  this  model  into  an  intelligent  cognitive  agent  was  also 
discussed,  in  an  effort  to  create  a  visual  representation. 

The  group  also  discussed  how  the  military  is  currently  implementing  technologies  that 
look  at  physiological  responses  to  interrogations,  and  how  a  simulation  can  capture  that.  One 
example  is  for  the  simulation  to  teach  culturally  specific  things.  The  group  also  discussed  how  to 
capture  the  entire  spectrum  of  language  and  communication,  including  non-verbal  body 
language,  paralinguistics,  and  neuro-linguistics  in  a  virtual  reality  trainer,  as  well  as  the 
implications  of  these  various  models,  and  the  potential  for  a  greater  level  of  detail  in  input.  It  is 
unknown  at  this  time  what  level  of  input  is  needed  to  teach  a  person  to  be  an  effective 
communicator  or  interrogator,  such  as  knowing  the  pitch  of  the  voice  or  the  heart  rate.  However, 
which  specific  nuances  are  that  of  a  novice  versus  an  expert,  are  known.  In  concluding,  the 
group  discussed  the  need  for  a  trainer  to  be  refined  when  better  technologies  come  along,  as  the 
technology  does  not  yet  exist.  Perhaps  a  beginner  should  see  a  simulation  trainer  first  and  this 
will  make  the  training  time  shorter  in  the  more  advanced  courses  with  one-on-one  instruction 
with  a  human.  The  group  ended  its  discussing  by  asking  what  types  of  capability  they  are  trying 
to  provide,  such  as  skills  in  cultural  agility  or  something  more  culture-specific.  It  was  decided 
they  are  working  on  building  an  adaptable  solider. 
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The  final  development  session,  Using  Cultural  Informants,  Cultural  Subject  Matter- 
Experts  (SMEs),  and  Military  SMEs  in  Content  Development  and  Validation ,  began  with  a 
briefing  by  Dr.  Lewis  Johnson  of  Alelo.  He  opened  the  session  by  discussing  ways  to  use 
cultural  infonnants  and  SMEs,  including  finding  the  operational  context  that  they  are  training  for 
and  how  it  constrains  in  training.  Alelo  is  figuring  out  what  kind  of  information  operators  will 
need  and  then  they  collect  and  develop  that  content.  He  noted  the  importance  of  using  SMEs  at 
all  stages  of  the  development  process.  One  problem  with  this  is  using  retired  military  SMEs  as 
the  information  might  no  longer  be  valid.  It  is  therefore  important  to  collect  all  the  pieces  of  data 
and  infonnation  and  to  then  triangulate  from  it  what  is  believed  to  be  the  most  accurate 
perception.  Some  people  have  varying  levels  of  meta-cultural  awareness  of  their  own  culture,  so 
accounts  can  definitely  vary  within  SMEs.  Alelo  prefers  to  meet  with  SMEs  on  an  individual 
basis,  so  that  they  can  delve  deeper  into  their  expertise.  They  are  also  worried  about 
contamination,  since  if  there  is  a  dominant  person,  most  of  the  content  is  going  to  come  from  that 
person.  That  is  why  they  are  typically  not  in  favor  of  focus  groups,  and  prefer  one-on-one 
interviews. 

The  final  portion  of  the  development  track  was  led  by  Dr.  Winston  Sieck  (Applied 
Research  Associates).  He  discussed  how  to  get  information  and  how  to  ensure  it  is  valid,  as  well 
as  what  aspects  of  culture  to  train,  as  that  detennines  which  methods  they  use  to  develop  the 
content.  There  are  a  lot  of  different  kinds  of  expertise,  necessary  for  creating  good  content,  and 
include  cultural  informants,  cultural  SMEs  and  military  SMEs.  The  challenge  is  figuring  out  how 
to  understand  what  is  going  on  in  the  heads  of  people  from  other  cultures.  Military  experts  of  a 
region  do  not  provide  the  same  information  as  locals  from  that  culture  do,  and  it  should  not  be 
assumed  that  they  provide  the  same  information,  as  each  brings  a  unique  kind  of  expertise. 
However,  he  does  believe  that  basic  competencies  of  a  culture  can  be  generalizable,  as  long  as 
people  are  aware  of  what  drives  specific  cultures.  To  do  this,  they  must  first  establish  evaluation 
criteria,  including  the  difference  between  what  a  novice  and  an  expert  has  in  their  skill  set  and 
then  conducts  a  gap  analysis. 

Sieck’s  research  uses  the  method  of  Cultural  Network  analysis  to  map  knowledge  in  a 
way  one  might  map  a  social  network.  The  group  discussed  perceived  values  and  thoughts  that 
are  populated  with  a  specific  concept  and  what  information  is  needed  from  an  expert.  This  led  to 
discussion  on  what  constitutes  a  cultural  expert  and  difficulties  in  accessing  such  experts. 

To  document  information  received  from  an  interview  and  to  balance  the  scope  of  the 
interview  to  ensure  you  receive  the  necessary  information  involves  recording  the  session  and  to 
create  a  series  of  prepared  questions.  However,  gaps  will  be  discovered  later  on,  but  ideally  one 
can  return  to  the  subject  matter  experts  or  seek  new  experts  to  assist  in  filling  gaps.  Other  ways 
to  mitigate  these  problems  is  to  have  two  interviewers,  and  to  use  a  very  methodological 
approach. 

One  person  noted  the  importance  of  using  outside  perspectives  as  they  will  be  attuned  to 
minor  differences  of  a  culture,  as  often  times,  people  are  unable  to  point  out  specific  things 
within  their  own  culture  because  they  lack  the  meta  cognition.  Narratives  are  an  effective 
approach  for  getting  information  out  of  people  who  do  not  have  an  ability  to  reflect  on  their  own 


21 


culture.  Dr.  Sieck  also  discussed  the  importance  of  having  multiple  subject  matter  experts  to 
cross-check  infonnation  and  to  best  understand  the  data.  It  is  important  to  be  reflective  and 
careful  about  one’s  own  assumptions  when  going  into  an  interview,  because  how  the  interviewer 
engages  a  SME  is  going  to  be  very  different.  Even  the  kinds  of  questions  that  one  comes  up  with 
will  be  very  different.  To  control  for  the  many  different  opinions  of  a  culture,  they  try  to  collect 
data  from  several  sources  to  determine  multiple  courses  of  action.  Because  cultures  are  diverse, 
they  do  not  seek  one  answer.  It  is  important  to  expose  people  to  multiple  courses  of  action  in  a 
given  scenario.  Dr.  Sieck  also  discussed  using  internet  surveys  to  see  how  teamwork  varies 
across  cultures.  One  dilemma  with  using  the  internet  is  that  remote  villages  do  not  have  the 
necessary  access.  This  is  where  one  can  use,  for  example,  Afghani  experts  and  natives  to  go  into 
villages  and  talk  with  people,  in  order  to  get  a  good  idea  of  the  differences  between  people  in  the 
city  and  people  in  villages.  Another  example  is  to  use  role  playing  to  describe  non-verbal 
communication,  such  as  how  far  apart  people  stand  in  a  foreign  culture.  This  approach  can  also 
help  elicit  information  as  any  expert  would  have  an  opinion  on  it.  It  is  important  to  develop  and 
share  tools  to  help  the  collection,  structuring,  analysis,  and  reanalysis  of  data. 

To  validate  the  accuracy  of  the  data,  they  return  to  the  source  of  the  data  and  to  the 
subject  matter  experts  a  second  time.  To  look  into  accrediting  training,  it  was  suggested  people 
look  at  whether  it  covers  the  materials  a  Marine  will  need  to  learn  and  whether  it  is  specific  to  a 
particular  region,  however;  it  was  noted  there  is  no  complete  answer  yet.  Language  training  has 
a  more  standardized  accreditation  process.  It  is  handled  mainly  on  making  sure  the  correct 
language  is  being  taught  for  the  given  region  and  whether  dialects  change  drastically  between 
villages.  Applied  Research  Associates  has  its  language  material  validated  by  multiple  sources 
and  considers  it  an  iterative  process.  The  goal  of  the  research  is  to  get  knowledge  into  the  field, 
but  also  to  build  cultural  sense  making.  This  helps  to  create  high  content-specific  knowledge  that 
is  focused  on  core  generalizable  skills  for  teaching  cultural  ability. 

Discussions  addressed  the  need  for  greater  authorability  of  scenarios  so  that  training  can 
be  disseminated  more  rapidly.  The  traditional  way  for  making  training  is  for  one  person  to 
provide  content  and  for  the  developer  to  turn  it  into  a  program.  What  is  needed  is  for  both 
training  and  programs  to  be  made  adaptable  to  revisions  in  content.  A  current  challenge  is  the 
ability  for  someone  to  build  a  tool  and  technique  that  can  collect  cultural  data  and  quickly  put  it 
into  training  that  can  be  accessed  by  Soldiers  throughout  the  world. 


Development  Track  Recommendations 

Apply  and  disseminate  research  from  the  cognitive  and  learning  sciences  to  the  development 
of  cultural  training. 

Cultural  training  development  often  focuses  on  the  cultural  content  without  sufficient 
consideration  of  the  learner’s  prior  experiences,  cognitive  limitations,  and  learning  processes. 
Research  is  needed  to  apply  relevant  to  the  sociocultural  and  interpersonal  content  domain  to 
insure  that  training  development  is  strongly  founded  in  instructional  design  principles. 
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Use  stories  and  narrative  from  Service  members  to  provide  realistic  and  relevant  training 
content. 

This  approach  provides  engaging  material  to  help  convey  cultural  information  in  the 
context  of  Service  missions  and  cultures.  Research  should  examine  the  role  of  narrative  in 
facilitating  cultural  learning. 

Provide  methods  and  resources  to  gather,  store,  and  validate  cultural  data. 

Participants  suggested  that  sharing  data  would  help  facilitate  the  development  of  cultural 
models  and  of  instructional  material.  These  resources  would  include  data  on  specific  cultures  of 
interest,  as  well  as  critical  incidents  from  military  personnel  regarding  cultural  challenges  they 
encounter  in  their  missions.  Participants  recognized  a  need  to  insure  that  training  content  does 
not  reflect  the  biases  of  informants  who  may  have  incomplete,  inaccurate,  or  outdated 
information.  Data  sharing  and  common  validation  methods  would  help  reduce  the  risks 
associated  with  biased  or  limited  sources. 


Implementation/Evaluation  Track  Summary 

The  first  session  of  the  Implementation  and  Evaluation  track  was  led  by  Dr.  Joan  Rentsch 
(University  of  Tennessee,  Knoxville)  and  focused  on  identifying  skills  necessary  for  intercultural 
competency.  These  skills  include  interpersonal  skills  and  self-awareness,  but  also  include  a 
strong  knowledge  component.  Dr.  Rentsch  is  conducting  research  to  identify  cultural  knowledge 
that  is  useful  to  Soldiers.  The  goal  is  to  identify  a  schema  for  understanding  cultural  and  regional 
expertise.  In  order  to  do  this  and  to  understand  a  culture,  it  is  important  to  understand  how 
people  organize  knowledge.  These  knowledge  structures,  or  schemas,  are  formed  through  both 
direct  and  indirect  experiences  and  should  therefore  be  trainable.  Expert  schemas  serve  as  a  good 
target  for  knowledge  development.  Dr.  Rentsch  discussed  representing  and  extracting  complex 
cognitive  information  using  the  method  of  Paired  Comparison  Ratings.  Research  participants  rate 
the  relatedness  of  concept  pairs.  Once  these  pairs  are  created,  the  data  is  mapped  using  the 
Pathfinder  program,  which  is  based  on  graph  theory  and  which  represents  data  as  a  network  to 
show  relatedness.  The  benefit  of  this  model  is  that  it  shows  central  concepts  that  influence  other 
concepts.  An  example  application  is  that  the  schema  could  be  used  to  evaluate  training.  If 
students  are  asked  questions  and  the  answers  fit  the  models,  then  there  is  success. 

Questions  are  formulated  about  the  pairs  by  asking  what  the  two  items  in  the  pair  say 
about  culture,  and  how  they  link  to  the  abstract  notion  of  each  culture  rather  than  each  other.  Dr. 
Rentsch  worked  with  non-academic  subject  matter  experts  who  had  professional  experience  in  a 
military  environment.  They  were  interviewed  and  their  unique  statements  were  taken,  placed  on 
cards,  and  then  people  stacked  the  cards  as  they  thought  that  the  statements  were  related.  Then 
they  pulled  pairs  from  those  cards.  It  was  noted  that  the  Anny  could  use  this  process  of  taking 
two  weeks  to  collect  data,  create  a  model,  then  alter  the  training  methods  based  on  this  model  to 
test  its  effectiveness.  One  person  noted  that  he  does  something  similar  but  creates  his  own 
unique  terms  so  that  it  is  a  self-contained  system,  also  using  Pathfinder. 
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In  this  research,  rather  than  create  a  framework  of  culture,  the  goal  is  to  map  a  mental 
framework  of  how  an  expert  understands  culture.  Interviewees  were  Soldiers  who  had  been 
previously  deployed  to  multiple  foreign  countries.  Respondents  who  lacked  either  consistency  or 
coherence  in  their  data  were  dropped.  The  goal  is  to  identify  an  umbrella  schema  that  represents 
a  strong  operational  understanding  of  culture. 

The  second  presentation  was  by  Dr.  Peter  Siska  (USMA),  who  discussed  how  to 
understand  regional  knowledge,  using  regional  expertise  as  a  military  requirement.  He  focused 
on  five  major  components  of  regional  understanding  using  the  Pentathlete  concept.  Each  person 
has  a  personal  definition  of  culture  and  these  definitions  need  to  be  plotted  to  all  aspects  of 
regional  complexity.  To  assess  knowledge  and  competence,  officers  need  to  abstract  an 
understanding  of  a  general  region  and  then  apply  it  to  specific  regions.  Analytical  and  synthetic 
thinking  differ,  both  of  which  are  necessary.  Dr.  Siska  also  discussed  chorology,  a  cause  and 
effect  relationship,  which  keeps  the  regional  building  blocks  together,  such  as  politics  and 
economics,  for  example,  where  one  thing  is  dependent  on  another.  He  then  discussed  the 
General  Regional  Aptitude  Network  Test  (GRANT),  which  is  a  questionnaire  assessing  general 
regional  knowledge  and  motivation  for  regional  learning.  This  research  could  help  identify 
cadets  who  have  an  aptitude  for  regional  learning. 

Some  discussion  focused  on  geography  being  a  critical  skill  in  this  area.  Recent  BAAs 
have  focused  on  geospatial  overlays  and  this  group  is  now  focusing  on  the  correlation  between 
geography  and  culture.  For  example,  geography  might  be  important  for  understanding 
differences  between  villages,  given  their  distance.  Mountainous  regions  greatly  differ  from  flat 
terrain  regions  in  terms  of  cultural  development.  Region  was  also  one  of  the  pairs  plotted  in  the 
previously  discussed  network.  One  attendee  commented  how  the  Nazi  regime  was  deterministic 
in  its  approach  to  cultural  success  and  geography. 

Dr.  Paula  Durlach  (U.S.  Army  Research  Institute)  presented  the  third  session  on 
Designing  and  Conducting  Training  Evaluation.  In  Kirkpatrick’s  framework,  there  are  five 
levels  of  training  evaluation:  reaction,  learning,  behavior,  transfer,  and  return  on  investment,  the 
measure  of  impact  on  an  organization.  The  difficulty  in  assessing  increases  from  the  former  to 
the  latter.  At  each  step  it  becomes  harder  to  determine  how,  what,  and  when  one  should  measure. 
Methods  used  include  analyzing  the  differences  between  pre-  and  post-tests  and  comparing 
groups  that  receive  different  treatments,  such  as  measuring  a  group  receiving  several  series  of 
training,  which  is  good  for  assessing  different  pedagogical  models.  However,  this  method  is 
difficult  to  conduct.  The  ultimate  goal  of  training  is  for  the  trainee  to  use  what  is  learned  on  the 
job,  and  the  difficulty  resides  in  retention  and  application  of  what  is  learned.  To  measure  skills 
retention,  one  must  look  at  detecting  changes  in  knowledge  and  behavior,  which  includes  using 
increments  and  not  simple  yes  and  no  answers  from  respondents.  This  helps  ensure  the 
assessment  accurately  portrays  what  was  learned.  For  example,  it  is  important  to  measure 
aptitude,  motivation,  experience,  and  time,  in  addition  to  just  knowledge.  It  is  also  imperative 
that  the  purpose  of  the  training  evaluation  be  known,  since  policymakers  need  one  level  of  data 
and  academics  require  another. 
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Dr.  Durlach  then  discussed  BiLAT,  a  game-based  training  which  teaches  bilateral 
negotiation  in  an  Iraqi  cultural  context.  Assessment  used  a  type  of  situation  judgment  test  (SJT), 
which  measures  something  between  learning  and  associated  behavior.  Dr.  Durlach  discussed  SJT 
situations,  and  how  they  were  generated  and  vetted  with  subject  matter  experts  before  being 
revised  and  then  reviewed  by  a  second  set  of  subject  matter  experts.  At  that  point,  experts  raised 
questions  they  could  not  agree  on.  In  hindsight,  Dr.  Durlach  noted,  this  did  not  test  the  sensitivity 
needed  to  know  the  level  of  experience  of  the  test-taker.  In  examining  the  evaluation  and  design 
results,  there  was  a  significant  increase  in  post-test  scores  compared  to  pre-test  scores.  One  could 
also  see  a  correlation  between  the  knowledge  of  Iraqi  culture  and  the  level  of  experience  in 
negotiation.  People  who  had  negotiation  experience  started  out  high,  but  did  not  increase  their 
scores.  Additional  lessons  learned  included  the  need  to  understand  the  degree  of  sensitivity  prior 
to  the  testing  and  knowing  exactly  what  one  is  measuring.  Finally,  she  concluded  that  evaluation 
should  not  be  the  last  step,  but  rather  a  part  of  a  continuous  evaluation. 

Ms.  Amy  Ogan  (Carnegie  Mellon  University)  discussed  human-computer  interaction, 
also  using  BiLAT.  Characters  were  created  based  on  experiences  of  US  Soldiers  in  the  field. 

The  learning  objective  is  to  establish  relationships  and  to  create  win/win  situations.  Certain  items 
are  measurable  like  steps  for  a  proper  way  to  pray  in  a  foreign  culture.  It  is  important  to  identify 
such  practices  as  they  can  lead  to  the  best  possible  social  outcomes.  She  noted  that  students  treat 
BiLAT  like  a  game  with  no  consequences. 

Ms.  Ogan  conducted  a  randomized  study  in  which  students  who  were  given  social  goals 
for  the  game  performed  better  than  students  without  goals.  Dependent  variables  included 
retention,  timing,  and  matching  learning  with  study  objectives.  In  the  assessment,  she  looked  at 
independent  variables,  comparison  conditions,  and  versions  of  simulation.  At  least  twenty 
participants  were  needed  in  every  condition  to  have  statistical  power,  and  they  also  needed  an 
intervention  time  to  avoid  fatigue.  To  collect  data,  she  avoided  experimenter  bias  by  writing  a 
script  that  was  constant  to  both  conditions.  BiLAT  had  a  ceiling  effect  on  both  pre-  and  post¬ 
tests  with  the  control  group  having  non-social  goals  performing  better  in  some  areas.  In 
conclusion,  there  was  no  difference  between  the  two  groups  in  terms  of  personal  goals,  but 
students  with  social  goals  took  fewer  actions. 

During  the  discussion  period,  attendees  discussed  training  objectives  of  the  BiLAT 
program,  which  establishes  trust  and  uses  a  win/win  strategy  to  train  bilateral  engagement.  These 
are  high-level  objectives  drilled  into  each  skill-set,  though  some  attendees  felt  these  are  vague 
and  difficult  to  measure.  They  also  have  a  list  of  save-face  options  in  the  SJT  and  the  student 
determines  which  is  best.  There  was  also  a  discussion  on  knowing  what  is  and  is  not  measurable. 
It  can  be  difficult  to  find  quantitative  observable  factors.  It  was  suggested  that  the  Services  create 
ten  steps  to  learning  negotiation  skills.  However,  it  was  noted  that  social  skills  can  be  difficult  to 
define. 


Additionally,  the  group  discussed  a  video  shown  to  the  group  the  prior  day.  They 
discussed  how  this  video  failed  to  show  respect  and  the  importance  of  teaching  Service  members 
how  to  think  in  addition  to  how  to  behave.  They  also  discussed  how  evaluation  is  conducted, 
noting  the  need  for  independent,  third-party  evaluation,  as  opposed  to  training  developers 
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conducting  evaluation  on  their  own  products.  It  is  helpful  to  triangulate  this  to  identify  which 
areas  are  working  versus  not  working,  and  to  look  at  the  same  problem  from  various  angles. 
BiLAT  welcomes  assistance  in  creating  its  evaluation  process  to  better  aid  them  in  understanding 
their  system.  Currently,  they  use  military  SMEs  and  Iraqis  who  are  professional  role-players  to 
inform  their  systems.  They  are  beginning  to  also  look  at  what  Iraqis  who  do  not  have  military 
training  are  thinking. 

The  group  concluded  with  a  conversation  on  the  role  of  SMEs,  including  who  they  are, 
what  they  know,  and  what  really  makes  them  expert,  especially  those  in  the  military.  Some 
people  noted  the  importance  of  working  with  the  Department  of  State,  who  conducts  language 
training  for  diplomats. 

In  a  third  session,  Dr.  Melissa  Brittain  (Air  Force  Culture  and  Language  Center)  and  Dr. 
Joyce  Osland  (San  Jose  State  University)  led  presentations  on  the  assessment  of  non-cognitive 
learning  outcomes.  Dr.  Brittain  discussed  Air  Force-specific  challenges  in  developing  and 
assessing  cross-cultural  competence,  and  ways  to  reinforce  cultural  learning  throughout  an 
Ainnan’s  career.  Ms.  Britton  focused  on  the  Air  Force’s  three  critical  culture  skills:  relate, 
communicate,  negotiate,  and  how  this  must  begin  in  a  classroom.  Education  on  culture  is 
currently  based  around  the  Quality  Enhancement  Plan  (QEP),  which  is  needed  to  ensure  Air 
Force  University  accreditation.  Dr.  Brittain  is  charged  with  four  student  learning  outcomes: 
declarative  knowledge,  skills  (communication,  negotiation,  relation),  attitude  (openness  to  other 
cultures),  and  application  (how  to  perform  in  the  field).  She  noted  the  difficulty  of  determining 
whether  these  learning  outcomes  enable  better  perfonnance  in  deployment;  for  example,  how 
might  cultural  relativism  relate  to  performance  and  how  much  is  appropriate? 

The  group  discussed  this  briefing,  including  how  to  assess  application,  which  is  done 
using  a  self-report  system  at  six  month  intervals.  They  are  also  working  on  behavioral 
observation  and  behavioral  diaries  to  develop  new  tools.  They  also  focus  on  broad  cultural  skills, 
such  as  the  basics  for  gathering  information  instead  of  regional  learning. 

Dr.  Brittain  also  discussed  the  Air  Force’s  instruction  approach,  which  includes  distance 
learning  courses.  One  person  questioned  whether  Airmen  are  actually  utilizing  the  skills  they 
have  learned.  Ainnen  receive  cultural  instruction  both  in  education  and  in  expeditionary  training 
prior  to  deployment,  and  they  receive  both  cultural  general  and  cultural  specific  instruction.  They 
are  also  provided  with  simulations  to  reinforce  their  training.  They  are  currently  working  to 
figure  out  what  works  best  at  changing  one’s  attitude,  including  stimulus,  reflection,  and 
evaluation. 

The  group  also  discussed  how  a  warfighter  might  forget  about  his  cultural  training  when 
he  is  in  a  traumatic  cross-cultural  experience.  In  addition,  confidence  in  one’s  cultural  skills  is 
probably  helpful  to  an  extent,  but  the  group  also  saw  value  in  failure  experiences  that  provide  a 
realistic  view  of  one’s  knowledge  and  skill  levels.  Other  challenges  include  testing  and 
evaluation  with  such  large  numbers  of  personnel,  as  well  as  how  to  assess  at  higher  ranks  and 
levels  of  competence,  and  what  critical  non-cognitive  attributes  should  be  addressed. 
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It  was  also  noted  that  Service-specific  deployment  models  may  impact  the  learning 
outcomes  of  interest.  Whereas  the  Anny  and  Marine  Corps  tend  to  train  and  fight  as  a  single 
unit,  the  Air  Force  deploys  individuals,  so  they  are  training  broad  behavioral  indicators.  Adding 
to  Service  differences  is  the  fact  the  Air  Force  has  a  QEP  guideline,  which  is  education  focused 
and  not  task-  or  mission-oriented.  One  recommendation  is  to  have  J3  fund  cross-cultural 
education  to  provide  more  consistency  across  Services  in  baseline  knowledge  and  skill. 

Next  the  group  discussed  online  courses  and  whether  an  instructor’s  sample  of  behavior 
is  enough  to  assess  attitude.  The  Air  Force  requires  attitude  assessment  throughout  the  course, 
but  they  need  a  way  to  track  the  development  as  the  current  rubric  is  not  the  best  way.  It  is 
important  to  have  a  positive  attitude  so  that  the  trainee  can  adopt  behaviors  to  achieve  an 
effective  outcome.  Predispositions  and  base  attitudes  are  often  resorted  to  during  times  of  stress. 
The  group  discussed  the  internalization  of  training  across  the  services  and  the  need  to  change  the 
attitude  of  the  military.  However,  this  issue  was  a  point  of  controversy,  as  past  research  has 
shown  that  attitudes  are  often  not  a  good  predictor  of  behavior.  There  was  disagreement  among 
the  group  about  the  relative  importance  of  attitudinal  and  affective  learning  outcomes  versus 
behavioral  outcomes.  Participants  noted  that  front-end  analysis  might  help  address  some  of  these 
issues. 


The  fourth  session  of  the  Implementation/Evaluation  Track  focused  on  the  Impact  of 
Cultural  Training  on  Readiness  and  Mission  Performance,  led  by  Dr.  Jay  Goodwin  (U.S.  Army 
Research  Institute)  and  Dr.  Kurt  Kraiger  (Colorado  State  University).  Dr.  Kraiger  discussed  the 
use  of  training  evaluations,  and  whether  or  not  they  add  value.  He  noted  they  are  often  used 
because  money  is  being  invested,  and  the  investor  wants  to  know  what  is  working  well. 

However,  they  become  difficult  when  one  is  trying  to  use  them  to  make  decisions.  Most 
evaluation  measures  reactions,  and  only  a  few  measure  learning.  Dr.  Kraiger  discussed  the 
Kirkpatrick  levels  of  training  evaluation,  which  is  the  most  commonly  used  model  of  evaluation, 
but  noted  its  criticisms,  such  as  causally  relating,  and  making  the  assumption  that  if  a  student  did 
not  like  the  training,  then  he  did  not  learn.  It  is  important  to  ask  what  one  hopes  to  understand 
from  an  evaluation.  The  purpose  of  the  evaluation  must  be  clear:  to  make  decisions,  to  get 
feedback,  to  have  relevancy,  or  to  use  in  marketing.  The  value  of  measuring  is  in  knowing  what 
one  is  doing,  and  why.  The  briefing  concluded  with  a  discussion  on  the  ways  to  quantify  the 
effect  of  cultural  training  and  looking  at  its  long-tenn  benefits. 

Dr.  Jay  Goodwin  gave  the  final  briefing  of  the  Implementation  and  Evaluation  track  and 
discussed  program  effectiveness,  including  the  transfer  of  training,  individual  and  unit  readiness, 
and  practical  issues.  He  pointed  out  that  there  is  a  big  difference  between  behavioral  routines  and 
knowledge  and  skills.  Behavioral  routines  are  default  reactions  and  often  do  not  help  in  cross- 
cultural  situations,  and  therefore  some  basic  cultural  skills  are  needed.  The  training  audience 
must  be  intellectually  ready  to  make  use  of  these  skills.  The  military  defines  readiness  as  the 
ability  to  perform  a  task.  Culture  can  be  thought  of  as  the  context  of  a  task.  Time  and  technology 
are  two  issues  that  they  must  deal  with.  Cultural  training  is  an  umbrella  tenn.  Its  general  purpose 
is  to  both  serve  the  broad  mission  by  interacting  with  people  and  it  varies  at  the  military  level,  as 
well  as  interacting  with  locals. 
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One  person  in  the  group  felt  they  are  standardizing  too  much.  Dr.  Goodwin  responded 
that  the  military  is  still  in  the  early  stages  of  this  process  and  that  there  is  merit  in  some 
standardization  of  the  knowledge  and  skill  set  for  cultural  capabilities.  Nowadays,  ground  forces 
need  to  be  able  to  talk  to  locals,  and  not  just  shoot.  To  evaluate  the  aspect  of  time  measuring 
impact  decades  down  the  road,  the  military  must  gradually  perform  tighter  studies,  first  showing 
how  those  with  training  are  more  trusted,  and  then  show  those  who  are  more  trusted  being 
promoted. 

Another  gap  exists  in  how  to  measure  cultural  readiness.  The  Services  provide  reports 
each  month  that  show  how  a  unit  is  maintaining  skills.  Readiness  is  about  demonstrating 
performance,  and  training  is  about  perfonnance.  It  was  unknown  if  there  are  any  soft  human 
skills  in  current  readiness  reporting.  It  is  important  to  teach  more  than  just  language  skills.  The 
warfighter  must  also  have  the  skills  to  navigate  a  foreign  culture  with  little  or  no  foreign 
language  proficiency.  Anecdotes  about  the  effects  of  training,  which  are  taught  at  a  few  military 
training  centers,  can  be  just  as  important  as  numbers  and  perhaps  even  more  effective.  Anecdotes 
can  be  used  to  prove  value.  They  are  not  data,  but  data  can  be  gleaned  from  them.  It  is  important 
to  have  anecdotes  about  successes,  failures,  and  near-misses.  To  know  whether  an  anecdote  is 
successful  as  an  evaluation  tool,  one  must  consider  the  context  of  to  whom  it  is  being  provided, 
for  example,  the  military,  or  a  news  organization.  The  group  addressed  the  need  for  gathering 
and  disseminating  these  kinds  of  anecdotes.  Questions  were  raised  about  how  to  quantify  return 
on  investment  in  this  area;  this  issue  was  noted  as  one  area  where  much  work  is  needed. 


Implementation/Evaluation  Recommendations 

Develop  methods  to  evaluate  the  effects  of  cultural  training. 

There  is  currently  no  way  to  quantify  the  effect  of  cultural  training.  Research  should 
develop  measures  for  the  relevant  cultural  learning  outcomes  and  methods  to  study  the  return  on 
investment.  Evaluation  and  comparison  of  current  cultural  training  should  be  conducted  to 
determine  whether  cultural  training  is  achieved  the  desired  effect  and  to  identify  continuing 
training  gaps.  Training  evaluation  methods  would  help  both  the  Services  and  industry  training 
developers  to  build  evaluation  into  the  design  process. 

Use  independent  reviewers  in  evaluations. 

It  is  important  to  include  independent  reviewers  in  training  evaluation.  It  is  important  to 
assess  higher  levels  of  trainees  at  a  different  degree  than  beginners  and  to  look  at  critical  non- 
cognitive  attributes.  Subject  matter  experts  from  within  the  military  are  not  necessarily 
independent  enough  to  evaluate  training  alone.  It  is  vital  to  use  reviewers  from  across  the 
broader  community. 

Create  common  baseline  cultural  training  standards  across  the  Services. 

Participants  felt  that  the  Services  could  identify  a  set  of  common  cultural  learning 
outcomes/objectives  that  all  personnel  should  achieve.  Though  the  Services  will  differ  in  many 
ways  in  their  approach  to  cultural  training,  a  common  baseline  set  of  skills  and  knowledge  would 
help  institutionalize  culture  within  the  military  as  an  enabler  to  mission  success. 
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Research  Gaps  and  Recommendations 


Breakout  sessions  identified  gaps  and  recommendations  not  only  for  research,  but  also 
for  structuring  education,  training,  and  other  development  of  cultural  capabilities.  In  this  section, 
the  primary  emphasis  is  recommendations  for  research  to  support  cultural  training  and  education 
efforts.  This  section  presents  an  overall  synthesis  of  research  issues  and  recommendations  from 
the  breakout  tracks,  the  plenary  sessions,  and  informal  discussions  among  workshop  participants. 
Many  of  the  research  gaps  identified  in  the  workshop  have  been  previously  identified  as  issues  of 
interest  to  the  Service’s  culture  centers  and  other  organizations  involved  in  force  development  in 
the  sociocultural  domain. 

In  general,  the  gaps  and  recommendations  presented  here  relate  to  training  and  education 
where  culture  is  a  central  focus.  However,  social  and  cultural  learning  can  either  be  a  foreground 
or  a  background  for  force  development  efforts,  and  workshop  discussions  suggested  a  need  for 
both.  Cultural  learning  objectives  are  sometimes  the  primary  focus,  but  more  often,  mission 
skills  are  the  focus  in  a  cross-cultural  context  -  that  is,  culture  is  often  a  condition  for  training 
and  simulation,  not  a  task.  Finer  distinction  among  different  modes  of  cultural  instruction  would 
be  beneficial  to  discussions  on  this  topic.  Borrowing  from  intercultural  training  research 
(Batchelder,  1978),  some  suggested  terms  are  offered  here  for  clarification. 

What  is  often  referred  to  as  pre-deployment  cultural  training  would  be  more  accurately 
labeled  cultural  orientation.  Cultural  orientation  provides  knowledge  of  the  specific  cultural 
situation  that  personnel  will  face:  the  “who,”  “what,”  “when,  and  “where.”  This  type  of 
instruction  conveys  facts  about  a  culture  or  country  and  may  include  specific  actions  to  take  or 
avoid  when  working  in  that  culture. 

Cultural  training  consists  of  preparing  personnel  to  perform  specific  tasks  in  an 
intercultural  environment  -  learning  and  practicing  mission-oriented  skills  for  operations  with  a 
cultural  component.  Cultural  education  builds  the  foundational  knowledge,  affect,  and  skills  for 
mission-oriented  skills,  particularly  when  cultural  performance  demands  are  broad,  uncertain,  or 
are  expected  to  vary.  Whereas  cultural  education  provides  the  “why,”  cultural  training  provides 
the  “how”  (Batchelder,  1978;  Bennett,  1986). 

Simulation  and  mission  rehearsal  offer  opportunities  to  practice  mission-oriented  skills  in 
a  realistic  context,  often  treating  culture  as  a  condition  for  the  mission,  much  like  an  aspect  of  the 
physical  terrain.  In  simulation,  culture  is  the  background.  In  cultural  orientation  and  education, 
culture  is  the  foreground.  Training  can  involve  culture  in  either  way,  either  developing  skills  that 
directly  enable  one  to  deal  with  the  cultural  components  of  the  mission,  or  developing  skills  that 
for  which  the  cultural  context  is  merely  incidental. 

These  distinctions  are  important  because  they  make  different  contributions  to  cultural 
readiness.  Conducting  simulation  and  mission  rehearsal  with  a  cultural  component  assumes  that 
participants  have  already  had  opportunities  to  acquire  the  skills  needed.  Focusing  on  only  one  or 
two  modes  of  instruction  provides  an  incomplete  background,  risking  gaps  and  redundancies  in 
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servicemembers’  cultural  preparation.  All  modes  need  to  be  considered  as  part  of  a  continuum  of 
preparation,  though  the  role  of  each  may  differ  depending  on  the  Service,  mission,  or  personnel. 

Incorporating  cultural  conditions  into  simulation  and  mission  rehearsal  represents  its  own 
set  of  challenges,  but  workshop  discussions  tended  to  focus  on  explicitly  addressing  cultural 
learning  objectives  rather  than  treating  cultural  conditions  as  implicit  to  other  tasks.  As  a  result, 
the  gaps  identified  and  recommendations  outlined  below  apply  most  strongly  to  cultural  training, 
orientation,  and  education,  and  to  a  lesser  extent  apply  to  simulation  and  mission  rehearsal  as 
defined  above.  Six  areas  needing  future  investment  are  discussed  in  detail  below: 

1)  Analysis  of  performance  requirements, 

2)  Learner  motivation,  cross-cultural  competence,  and  development, 

3)  Instructional  sociocultural  content, 

4)  Flexible  instructional  solutions, 

5)  Methods  and  metrics  for  training  evaluation, 

6)  Continuing  opportunities  for  exchange  and  collaboration. 


1.  Analysis  of  Performance  Requirements 

Workshop  sessions  called  for  a  clear  identification  of  performance  goals  and  cultural 
learning  outcomes  for  general  purpose  forces  (GPF)  and  critical  specialist  occupations.  Where 
possible,  cognitive  task  analysis  should  be  conducted  to  determine  perfonnance  requirements 
and  identify  gaps.  Specification  of  training  requirements  can  only  be  conducted  once  the 
performance  requirements  are  identified  and  other  potential  solutions  across  DOTMLPF 
(Doctrine,  Organization,  Training,  Materiel,  Leadership,  Personnel,  Facilities)  are  considered. 
Training  is  only  one  potential  means  to  filling  a  performance  gap.  Other  force  development 
activities  should  be  considered,  including  education  and  professional  assignment,  as  well  as  other 
personnel  and  organizational  solutions,  such  as  personnel  selection  and  assignment  and  the  use 
of  job  aids. 

To  date,  no  systematic  analysis  has  been  conducted  to  identify  the  human  perfonnance 
gaps  and  determine  which  solutions  are  most  appropriate.  As  a  result,  the  human  capabilities 
related  to  culture  have  not  been  specified,  and  the  means  to  achieving  them  have  not  been  clearly 
identified.  Developing  training  and  simulations  without  this  analysis  as  a  first  step  amounts  to 
guesswork  that  may  ultimately  prove  inadequate  to  fill  the  sociocultural  capabilities  gaps. 

Furthermore,  the  cultural  training  and  education  needs  for  specific  roles  and  functions  are 
poorly  understood.  Workshop  participants  argued  against  “one-size-fits-all”  cultural  instruction, 
but  we  do  not  yet  know  how  performance  requirements  and  training  needs  differ  by  job.  There 
may  be  a  baseline  of  knowledge  and  skill  that  all  personnel  need,  but  training  needs  will  likely 
diverge  through  the  career  progression.  Currently,  cultural  orientation  is  readily  available  for 
different  countries,  but  training  and  education  have  not  yet  been  tailored  for  specific  roles,  ranks, 
or  levels  of  expertise.  To  this  point,  training  and  simulation  development  has  focused  on 
incorporating  culture  for  meet-and-greet  interpersonal  interactions,  and  little  attention  has  been 
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directed  to  other  situations  and  roles  where  culture  is  relevant  (e.g.,  interrogation  and  intelligence 
collection,  managing  multiple  ongoing  relationships,  operational  planning). 

Research  should  first  specify  what  successful  perfonnance  in  this  domain  looks  like,  and 
identify  current  and  expected  future  perfonnance  gaps  for  different  roles  and  personnel.  Then  the 
human  capabilities  needed  to  achieve  the  desired  performance  levels  can  be  identified.  Those 
capabilities  will  provide  guidance  for  the  learning  outcomes  to  be  targeted  by  instructional 
interventions. 

2.  Learner  Motivation,  Cultural  Competence,  and  Development 

Discussions  identified  the  need  for  theory  advancement  in  two  areas:  the  cultural  content 
and  the  instructional  components.  The  cultural  content  is  addressed  in  a  subsequent  section.  For 
the  instructional  components,  frameworks  and  theory  on  cultural  learning  to  guide  instructional 
design  are  lacking.  Research  on  cultural  skill  acquisition  and  the  development  of  cultural 
expertise  is  greatly  needed.  There  is  a  large  literature  on  novice-expert  differences,  but  this 
research  has  not  been  widely  applied  to  the  development  of  cultural  expertise.  In  addition, 
research  is  needed  to  test  theory  on  the  development  of  cross-cultural  competence  and  other 
cultural  capabilities.  Theory  and  frameworks  for  intercultural  competencies  and  attributes  have 
been  outlined  both  for  military  (Abbe,  Gulick,  &  Hennan,  2007;  Hardison,  Sims,  Ali,  Villamizar, 
Mundell,  &  Howe,  2009)  and  non-military  populations  (Black,  Mendenhall,  &  Oddou,  1991; 
Johnson,  Lenartowicz,  &  Apud,  2006),  but  none  have  been  tested  for  their  ability  to  predict 
performance  in  military  personnel.  Generating  appropriate  content,  development  through  a  career 
progression,  and  tailoring  instruction  to  the  individual  all  depend  on  an  underlying  conceptual 
model  for  acquiring  cultural  knowledge  and  skills.  Research  to  identify  how  this  development 
occurs  will  better  enable  the  Services  to  close  the  gap  between  current  and  desired  perfonnance 
levels. 


Research  on  the  learner  population  is  needed  to  support  decisions  on  when  to  use 
different  instructional  methods  and  media.  Some  research  gaps  are  unique  to  the  domain  of 
cultural  learning,  whereas  others  relate  to  learning  and  instruction  more  broadly.  For  example, 
much  attention  has  been  paid  to  generational  differences  in  learning  and  motivation  (cf.  Prensky, 
2005),  differences  that  are  often  cited  in  discussions  on  the  use  of  games  in  training  and 
education.  However,  some  skepticism  may  be  warranted.  Empirical  evidence  has  repeatedly 
suggested  that  these  trends  are  overestimated  (Orvis,  Horn,  &  Belanich,  2006;  Pashler, 
McDaniel,  Rohrer,  &  Bjork,  2009;  Trzesniewski  &  Donnellan,  2009).  Some  generational  or 
individual  differences  may  be  very  important  (e.g.,  levels  of  experience  and  expertise  in  the 
subject  matter),  and  others  may  have  little  impact.  Further  empirical  research  is  needed  to  insure 
that  the  instructional  methods  and  media  selected  are  effective  and  efficient  for  the  target 
audience. 

One  important  topic  for  future  research  is  learner  motivation.  Recognizing  that  the 
relevance  of  cultural  instruction  should  be  highlighted  for  the  learner  (the  “so  what”  of 
instruction),  there  remain  a  set  of  unresolved  issues  surrounding  learner  motivation.  How 
important  is  the  role  of  “fun”  in  learning  in  games  and  simulation?  The  notion  of  fun  in  synthetic 
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learning  environment  seems  to  revolve  around  learner  engagement.  Though  unengaged  and 
inattentive  learners  will  almost  certainly  learn  less  than  the  engaged  and  attentive,  it  is  unclear 
whether  fun  and  enjoyment  is  a  requirement. 

Another  potentially  important  aspect  of  learner  motivation  is  the  distinction  between 
performance  motivation  and  mastery  motivation  (Elliott  &  Dweck,  1988).  When  perfonnance 
motivation  is  high,  the  learner  seeks  to  demonstrate  his  or  her  ability  to  perform  the  task.  When 
learning  motivation  is  high,  the  learner  seeks  to  master  the  skills  needed  and  will  seek  out 
challenges  that  provide  opportunities  for  skill  improvement  (Ford,  Smith,  Wessbein,  Gully,  & 
Salas,  1998).  These  goal  orientations  are  important  considerations  in  the  design  of  instructional 
interventions,  as  well  as  potentially  important  learning  outcomes  for  those  interventions  (Ford, 
Kraiger,  &  Merritt,  2010).  There  are  many  important  questions  surrounding  learner  motivation 
for  cultural  instruction.  For  example,  does  perfonnance  orientation  in  a  game  or  simulation 
undennine  learning?  How  can  synthetic  learning  environments  be  structured  to  enhance  learning 
and  mastery  goals? 

3.  Instructional  Sociocultural  Content 

One  potential  method  to  enhance  learner  engagement  and  motivation  is  the  use  of 
narrative,  scenarios,  and  stories,  which  were  viewed  by  workshop  participants  to  be  important 
resources  for  cultural  learning.  Some  of  the  general  benefits  of  scenarios  and  narrative  may  be 
higher  personal  relevance,  higher  learner  engagement,  and  utility  for  teaching  tacit  or  procedural 
knowledge  (vs.  declarative  knowledge).  Narrative  and  scenarios  may  be  particularly  well-suited 
for  developing  cultural  knowledge,  as  the  impact  of  culture  can  be  highly  situational  and  often 
occurs  below  conscious  awarenesss.  Research  is  needed  to  examine  these  issues  and  the  potential 
implications  for  learning.  Research  is  also  needed  to  determine  how  best  to  structure  cultural 
scenarios  for  effective  education  and  training.  A  bank  of  critical  incidents  related  to  culture  and 
intercultural  interactions  would  be  a  helpful  resource  for  training  developers,  as  well  as  a  helpful 
resource  in  determining  what  competencies  are  needed  in  these  situations.  Workshop  discussions 
cited  a  potential  role  for  the  Lessons  Learned  organizations  in  gathering  such  incidents,  which 
currently  lack  systematic  methods  to  collect  and  store  relevant  cultural  incidents. 

The  validation  of  cultural  content  in  cultural  orientation,  training,  and  simulation  is  of 
critical  importance.  Use  of  subject  matter  experts  (SMEs)  as  cultural  infonnants  during  content 
development  is  a  common  approach;  however,  the  term  “SME”  is  used  very  loosely  and 
imprecisely.  There  is  no  consensus  definition  of  a  cultural  SME,  and  SMEs  can  vary 
dramatically  in  the  recency,  breadth,  and  depth  of  their  experience  with  the  culture  in  question. 
Reliance  on  multiple  SMEs  is  one  solution,  but  combining  disparate  responses  is  still  a 
challenge.  Some  methods  are  available  to  determine  how  to  characterize  a  culture,  when  cultures 
are  imperfectly  shared  and  their  influence  situationally  activated  (e.g.,  cultural  consensus  theory, 
cultural  network  analysis).  Further  research  should  examine  these  methods  and  their  use  with 
different  sources  and  amounts  of  data. 

At  a  more  macro  level,  conceptual  models  and  taxonomies  are  needed  for  cultural 
differences  that  have  operational  relevance.  These  models  and  taxonomies  can  either  be  taught 
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directly  to  trainees,  as  guiding  cognitive  structures  for  cultural  knowledge,  or  can  serve  as 
ontologies  for  simulations  that  depict  different  configurations  of  cultural  values,  beliefs,  and 
behaviors.  Because  not  all  aspects  of  culture  are  equally  important  for  all  operations,  workshop 
participants  were  skeptical  that  existing  taxonomies,  such  as  those  developed  in  international 
business  settings,  would  apply  uniformly  to  military  operations.  A  better  understanding  of  how 
social  and  cultural  context  affects  behavior  in  conflict  situations  is  needed.  This  context  includes 
societal  or  national  culture,  but  also  includes  organizational  cultures  and  other  social  dynamics, 
such  as  those  due  to  military  intervention  itself. 

A  related  issue  is  detennining  the  appropriate  level  of  fidelity.  Cultural  content  can  be 
highly  accurate  without  being  relevant  or  important.  What  level  of  fidelity  is  needed  for  training 
and  simulation?  How  fine-grained  should  the  representation  be?  Fidelity  questions  can  only  be 
addressed  with  regard  to  the  operational  context  -  the  relevant  aspects  of  a  culture  are  not 
necessarily  the  same  across  all  missions  and  all  levels,  and  continuing  research  is  needed  to 
determine  what  social  and  cultural  infonnation  is  needed  to  perform  in  different  missions  and  in 
different  cultural  contexts. 

4.  Flexible  Instructional  Solutions 

One  set  of  research  challenges  relates  to  enhancing  the  flexibility  of  synthetic  learning 
environments  (i.e.,  training  simulations,  serious  games,  virtual  immersive  environments,  etc.). 
One  research  topic  to  support  that  flexilibity  is  the  identification  of  which  aspects  of  the 
operational  environment  are  important  for  learning,  and  therefore  which  aspects  are  important  to 
represent  in  the  synthetic  learning  environment.  Simply  representing  as  many  details  as  possible 
from  a  live  training  simulation  or  from  the  operational  environment  is  not  practical  and  probably 
not  necessary;  for  instructional  purposes,  selecting  the  details  with  direct  impact  on  learning  is 
the  primary  concern  and  likely  more  easily  accommodated. 

The  need  for  greater  authorability  in  training  and  simulation  was  also  recognized  as  an 
important  area  for  additional  research.  Instructors  would  benefit  from  greater  flexibility  to  update 
scenarios  based  on  feedback  from  the  field  and  changing  instructional  needs.  Trainers  and 
instructors  may  also  want  to  control  the  level  of  challenge  presented  in  a  scenario.  Current 
scripting  and  branching  does  not  allow  for  instructor-initiated  modification  without  considerable 
time  and  effort.  In  addition,  reusable  architectures  that  support  development  of  training  tools  for 
different  countries  or  regions  would  be  beneficial.  Participants  noted  a  need  for  tools  that  can  be 
edited  by  a  layperson,  not  a  programmer  or  other  specialist. 

Flexibility  is  also  needed  to  provide  learner-centered  instructional  design.  Current 
understanding  of  the  cultural  learning  domain  is  insufficient  to  support  tailoring  the  content  and 
level  of  difficulty  for  an  individual  learner.  Further  research  is  needed  to  determine  how  to 
address  different  levels  of  knowledge  and  skill,  how  feedback  should  be  structured,  and  what 
learning  experiences  help  an  individual  progress  from  novice  or  intermediate  skill  levels  to 
higher  levels  of  expertise.  Extensive  research  on  the  science  of  learning  has  informed  instruction 
for  other  content  domains;  this  body  of  knowledge  should  be  tested  in  and  applied  to  the  domain 
of  cultural  learning. 


33 


5.  Methods  and  Metrics  for  Training  Evaluation 

Methods  and  metrics  for  training  evaluation  are  greatly  needed.  As  in  other  sectors, 
gauging  user  reaction  is  common,  but  assessment  of  learning  is  relatively  rare  (Van  Buren  & 
Erskine,  2002).  Usability  and  user  reaction  are  infonnative,  but  measuring  learning  is  critical  and 
remains  a  continuing  gap.  Ideally,  assessments  of  learning  in  training  evaluation  would  be 
aligned  with  the  goals  for  cultural  learning  identified  in  doctrine  or  strategic  documents. 

Evidence  of  transfer  and  return  on  investment  would  also  be  ideal,  but  this  kind  of  research  is 
often  very  difficult  to  conduct.  Where  these  results  are  not  possible  to  assess,  research  should 
establish  a  link  between  mission-relevant  cultural  competencies,  learning  objectives  in  training 
and  education,  and  learning  outcomes  in  the  training  population.  This  insures  that  trainees  not 
only  learn  something  from  training,  but  that  what  is  being  learned  is  considered  important  and  is 
relevant  to  the  mission  -  in  other  words,  that  we  are  both  training  the  right  stuff  and  training  the 
stuff  right. 

Development  of  metrics  for  evaluation  of  learning  should  be  a  strong  focus  area  of  future 
research  on  cultural  training.  Any  training  development  efforts  should  build  evaluation  into  the 
design  and  development  process,  with  evaluation  metrics  designed  for  the  learning  objectives 
specific  to  that  training.  In  addition,  some  common  metrics  should  be  developed  to  enable 
comparisons  across  training  methods  and  products  that  address  similar  learning  objectives. 
Though  training  and  education  often  focus  on  cognitive  and  behavioral  learning,  workshop 
participants  recognized  the  need  for  further  research  on  the  full  range  of  relevant  learning 
outcomes  and  the  role  of  affect  and  metacognition  in  particular.  Different  learning  outcomes  may 
require  different  assessment  methods 

6.  Continuing  Opportunities  for  Exchange  and  Collaboration 

Though  defense  research  and  development  is  generally  conducted  to  address  a  problem 
for  a  particular  Service  or  organization,  it  should  also  contribute  to  the  broader  body  of 
knowledge.  Research  conducted  for  one  purpose  will  often  have  implications  for  other  questions, 
and  investigators  and  perfonners  should  be  encouraged  to  disseminate  their  findings  to 
appropriate  audiences.  Filing  reports  with  DTIC  is  helpful  but  insufficient,  as  project  reports  are 
often  intended  to  satisfy  with  requirements  of  a  contract  or  to  communicate  with  the  funding 
agency,  not  necessarily  to  communicate  with  others  working  on  similar  problems.  Research 
progress  occurs  by  sharing  findings  and  lessons  learned  in  an  incremental  fashion.  Indeed,  some 
initial  promising  work  has  been  conducted  on  many  of  the  topics  listed  above,  but  may  not  have 
reached  some  of  the  communities  that  would  benefit  from  it. 

HSCB  conferences  and  workshops  can  help  address  this  need,  and  such  events  should 
continue  on  a  regular  basis  to  allow  for  continuing  exchange.  In  particular,  there  is  a  need  to 
support  interdisciplinary  exchange.  Exchange  within  disciplines  occurs  in  many  existing 
professional  organizations.  However,  conducting  applied  research  on  such  broad  and  ill-defined 
problems  as  the  sociocultural  context  for  defense  activities  requires  interdisciplinary 
collaborations  that  are  often  not  readily  accommodated  in  these  organizations.  HSCB  and  other 
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defense  research  programs  can  play  a  central  role  in  facilitating  these  exchanges  through 
meetings  and  publications. 
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Appendix  A 


Workshop  Agenda 


Wednesday  4  November,  2009 

0730  Registration  &  Continental  Breakfast 

0830  Welcome  &  Opening  Remarks 

0900  Keynote  speaker:  Mr.  David  Ott,  representing  Mr.  Maxie  McFarland  (Deputy  Chief  of 

Staff  for  Intelligence,  U.S.  Training  and  Doctrine  Command) 

0945  Five  Principles  to  Consider  when  Developing  Education  and  Training  for  Cultural 

Knowledge  and  Skills 

Dr.  Richard  Clark  (University  of  Southern  California) 

1030  BREAK 

1045  Panel:  Perspectives  on  cultural  capability  needs 

Maj  Dov  Kawamoto,  Marine  Corps  Information  Operations  Center 

LtCol  Peter  Tarrant,  Defence  Centre  of  Training  Support,  UK 

1115  Culture  Centers  Panel  Part  I: 

Dr.  Paula  Holmes-Eber,  Marine  Corps  Center  for  Advanced  Operational  Culture  Learning 

LTC  Marc  Kortenray,  Training  Division,  Training  Development  and  Support 
Directorate,  U.S.  Army  Intelligence  Center 


1145  Lunch  Provided 
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1300  Breakout  Sessions 

Analysis:  Toward  a  shared  cultural  competency  map 

Design:  Designing  cultural  training  to  maximize  training  transfer 

Development:  Using  narrative  and  ethnographic  methods  in  cultural  instruction 

Implementation/Evaluation:  Assessing  non-cognitive  learning  outcomes 

1515  Break 

1530  Breakout  Sessions 

Analysis:  Motivational  and  personality  factors  in  cultural  learning 

Design:  Integrating  culture-specific  and  culture-general  objectives 

Development:  Cognitive  load  and  other  considerations  for  multimedia  delivery 

Implementation/Evaluation:  Impact  of  cultural  training  on  readiness  and  mission 
performance 

1730  Meeting  Adjourns 

1800  Welcome  reception 

Thursday  5  November,  2009 

0730  Registration  &  Continental  Breakfast 

0830  General  Session 

Meeting  the  cultural  training  and  simulation  needs  of  the  future  force 
Mr.  Martin  Bushika,  Marine  Corps  Systems  Command,  PM  Training  Systems 
0900  Culture  Centers  Panel  Part  II 

Dr.  David  Brand,  John  F.  Kennedy  Special  Warfare  Center  and  School 
Dr.  Rob  Sands,  Air  Force  Culture  and  Language  Center 
0930  BREAK 

0945  Breakout  Sessions 
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Analysis:  Sociocultural  aspects  of  mission  performance  in  specific  roles  and  MOS 


1200 

1315 


1530 

1545 

1630 

1730 


Design:  Gaming  and  simulation  for  intercultural  communication 

Development:  Theoretical  foundations  for  developing  virtual  humans  for  training  and 
simulation 

Implementation/Evaluation:  Assessing  regional  and  cultural  knowledge 
Lunch  Provided 
Breakout  Sessions 

Analysis:  Cultural  learning  as  a  developmental  progression 

Design:  Translating  cultural  competencies  into  learning  objectives  based  on  mission  and 
role 

Development:  Using  cultural  informants,  cultural  SMEs,  and  military  SMEs  in  content 
development  and  validation 

Implementation/Evaluation:  Designing  and  conducting  training  evaluation 
BREAK 

General  session:  Dr.  Dexter  Fletcher  (Institute  for  Defense  Analyses) 

Outbriefs  and  Wrap-up 
Meeting  Adjourns 
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Guide  to  Breakout  Sessions 


Wednesday  4  November  2009 
1300 


Track 

Topic 

Session  Leads  and  Presenters 

Analysis 

Toward  a  shared  cultural  competency  map 

Mr.  Marc  Hill 
(Defense  Language  Office) 

Mr.  Doug  Nelson 
(Kinection) 

Design 

Designing  cultural  training  to  maximize  training  transfer 

Dr.  Michelle  Wisecarver 
(PDRI) 

Mr.  Gonzalo  Ferro 
(PDRI) 

Development 

Using  narrative  and  ethnographic  methods  in  cultural 

instruction 

Dr.  Rob  Sands 
(U.S.  Air  Force 

Culture  and  Language  Center) 

Ms.  LeeEllen  Friedland 
(Ethnological 

Implementation/ 

Evaluation 

Assessing  non-cognitive  learning  outcomes 

Dr.  Melissa  Brittain 
(U.S.  Air  Force 

Culture  and  Language  Center) 

Dr.  Joyce  Osland 
(San  Jose  State  University) 

1530 


Track 

Topic 

Session  Leads  and  Presenters 

Analysis 

Motivational  and  personality  factors  in  cultural  learning 

Dr.  Joan  Johnston 

(Naval  Air  Warfare  Center  Training 

Systems  Division) 

Dr.  Paula  Caligiuri 
(Rutgers  University) 

Design 

Integrating  culture-specific  and  culture-general  objectives 

Dr.  Rick  Wolfel 
(Center  for  Language, 

Culture,  and  Regional  Studies, 

U.S.  Military  Academy) 

Dr.  Mark  Mendenhall 
(University  of  Tennessee,  Chattanooga) 

Development 

Cognitive  load  and  other  considerations  for  multimedia 
delivery 

Mr.  Gary  Rauchfuss 
(U.S.  Army  Training  and  Doctrine 
Command  G-3/5/7) 

Dr.  Chad  Lane 

(Institute  for  Creative  Technologies) 

Implementation/ 

Evaluation 

Impact  of  cultural  training  on  readiness  and  mission 
performance 

Dr.  Kurt  Kraiger 
(Colorado  State  University) 

Dr.  Jay  Goodwin 

(U.S.  Army  Research  Institute) 

Thursday  5  November  2009 
0945 
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Track 

Topic 

Session  Leads  and  Presenters 

Analysis 

Sociocultural  aspects  of  mission  performance  in  specific 

roles  and  MOS 

Dr.  Michelle  Zbylut 
(U.S.  Army  Research  Institute) 

Dr.  Karol  Ross 
(Cognitive  Performance  Group) 

Design 

Gaming  and  simulation  for  intercultural  communication 

Dr.  Elaine  Raybourn 
(Center  for  Language,  Culture,  and 
Regional  Studies,  U.S.  Military 

Academy) 

Dr.  Mike  Van  Lent 
(Soar  Technologies) 

Development 

Theoretical  foundations  for  developing  virtual  humans  for 
training  and  simulation 

Ms.  Julia  Kim 

(Institute  for  Creative  Technologies) 

Dr.  Lynn  Smith-Lovin 

(Duke  University) 

Implementation/ 

Evaluation 

Assessing  regional  and  cultural  knowledge 

Dr.  Peter  Siska 
(Center  for  Language, 

Culture,  and  Regional  Studies, 

U.S.  Military  Academy) 

Dr.  Joan  Rentsch 
(University  of  Tennessee,  Knoxville) 

1315 


Track 

Topic 

Session  Leads  and  Presenters 

Analysis 

Cultural  learning  as  a  developmental  progression 

Dr.  Melissa  Brittain 

(U.S.  Air  Force  Culture  and  Language 

Center) 

Mr.  Mike  McCloskey 
(361  Interactive) 

Design 

Translating  cultural  competencies  into  learning  objectives 
based  on  mission  and  role 

Dr.  David  Matsumoto 
(San  Francisco  State  University) 

Mr.  Doug  Nelson 
(Kinection) 

Development 

Using  cultural  informants,  cultural  SMEs,  and  military 

SMEs  in  content  development  and  validation 

Dr.  Lewis  Johnson 
(Alelo) 

Dr.  Winston  Sieck 
(Applied  Research  Associates) 

Implementation/ 

Evaluation 

Designing  and  conducting  training  evaluation 

Dr.  Paula  Durlach 
(U.S.  Army  Research  Institute) 

Ms.  Amy  Ogan 
(Carnegie  Mellon  University) 
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